Skip to content
 

UX issues around voting

While Andrew’s worrying about how to measure calibration and sharpness on small N probabilistic predictions, let’s consider some computer and cognitive science issues around voting.

How well do elections measure individual voter intent?

What is the probability that a voter who tries to vote has their intended votes across the ballot registered? Spoiler alert. It’s not 100%.

We also want to know if the probability of having your vote recorded depends on the vote. Or on the voter. To put it in traditional statistical terms, if we think of the actual vote count as an estimate of voter intent, what is the error and bias of the estimator?

Not very well, it turns out

User interface (UX) errors are non-negligible. For a concrete analysis around the 2000 U.S. presidential election, see the following page summarizing some of the findings of the

a big study partly coordinated by our collaborator Steve Ansolabehere.*

No surprise here

There’s nothing at all surprising here from a UX point of view. Everyone who’s ever worked on UX knows that UX errors are the norm, not the exception. The point of building a good UX is to minimize errors to the extent possible.

I also want to point out that banks seem to manage handing out cash through their ATMs with low enough error that it’s still profitable. Now I’m curious about the error rate.

Simple Example

In case you don’t want to click through to see the real example, an example of a classic UX blunder in a voting context is the following ballot.

[ ] CANDIDATE 1 [ ] CANDIDATE 2 [ ] CANDIDATE 3

This kind of layout violates the basic UX principle of putting checkboxes distinctively closer to their associated items than to other items. With the layout above, a voter who intends to vote for candidate 2 might accidentally vote for candidate 3 because the two boxes are equally close to the name “CANDIDATE 2”.

It’s better with more whitespace so that the boxes are visually identified with their associated item.

[ ] CANDIDATE 1      [ ] CANDIDATE 2      [ ] CANDIDATE 3

A vertical layout can solve some problems, but as the example in the article I linked above shows, it can introduce other ones if done poorly.

This is just one of the many blunders that are quite common in user interfaces.

Anecdote about my own ballot this year

Personally, I had a question about the fine print of the NYC ballots because there were a bunch of judge candidates and it wasn’t clear to me how many I could vote for. I actually flipped to the instructions, which said the number would be at the top. It wasn’t. I went and asked the poll worker out of curiousity (I’m fascinated by UX issues). Turns out they moved the number to relatively fine print to the left of the column of checkboxes. Now this particular vote didn’t matter as far as I can tell because there were only four candidates and you could choose up to four. Just an example of the kind of confusion you can run into.


* Steve’s the one who introduced me to Andrew 25 years ago after Andrew and I both moved to NYC. The second-to-last grant Andrew and I got before I moved to the Flatiron Institute was with Steve to work on his Cooperative Congressional Election Study data collection and analysis. That project’s still ongoing and the data, models, etc. are all open access.

44 Comments

  1. Anoneuoid says:

    Thanks.

    All of this comes down to a question of what error rate we can accept, although it may not be politically acceptable to admit that any errors can be predicted. There were over 29,000 invalid votes in Palm Beach County – more than 6% of the total. Nationwide, we routinely accept error rates of 3% – 4%. Why? Simply because those numbers would not affect the outcome of the race. What made the 2000 race so fascinating was that it was one of those rare occurrences when we were close to a statistical dead heat, given the error rates.

    Seems way too high to me.

  2. Royce says:

    > “No surprise here”

    Well, most Americans would be very surprised that our formal electoral systems have inherent margins of error due to their basic design and real world implementations.

    People incorrectly assume that official electoral processes are like counting colored marbles in a barrel — that there is only one correct, invariable count/sort of the marbles — and it always can be readily determined through simple counting, aggregation, and reporting procedures.

  3. Mikhail Shubin says:

    I guess it is a correct post to recall the “Japanese Instrument of Surrender” incident

    https://en.wikipedia.org/wiki/Japanese_Instrument_of_Surrender#/media/File:Japan_Instrument_of_Surrender_2_September_1945.jpg

    The Canadian representative, Colonel Lawrence Moore Cosgrave, signed below his line instead of above it on the Japanese copy, so everyone after him had to sign one line below the intended one.

    According To wikipedia, “This was attributed to Col. Cosgrave being blind in one eye” but I would say its just bad UX! Im frequently doing the same kind of mistakes.

  4. Joseph Schwarzbach says:

    May I ask if you had an opportunity to review the UX aspect of ranked-choice ballots?

    Indeed you’re talking about “capturing the voter’s intent” and there are good arguments in favor of trying to capture a richer view of the voter’s intent by letting them rank candidates – leading to fairer outcomes in most cases.

    From what I saw it appears that these ballots are *not* more confusing than regular ones.

    I’m kind of “plugging” ranked-choice voting into the discussion here, but really it has a UX dimension: I find it terribly frustrating when we set out to capture voters’ preferences at great cost and yet, let them express only a narrow view of their intent – their first choice – rather than capturing the wider view.

  5. Mikhail Shubin says:

    Speaking about UX and ballots, I wonder if the strategy used in Hitler’s ballot actually worked, and how many votes Hitler would have lost if a regular ballot was used.

    https://upload.wikimedia.org/wikipedia/commons/5/57/Stimmzettel-Anschluss.jpg

  6. expr says:

    On my ballot in NYS, for each position, the candidates were in a column with each party on a different (same for all positions) line
    which sort of implied that you vote for one in each column. In the judge elections you were supposed to choose 4. I wonder if people thought
    you had to pick one from each column.
    On my ballot it made no difference since for each judge column the same name appeared on each row
    You mark the ballot by filling in ovals and they feed it through a scanner
    Presumably if there is something that makes the ballot invalid, the scanner would show an error EXCEPT it does not flag positons fro which you did not vote for anyone so possibly incompletely filled bubbles could be missed

  7. Dale Lehman says:

    This is probably well known to most people here, but in case anyone is young enough (or so old that they forgot), we have the infamous “butterfly ballot” that might have been responsible for the results of the 2000 Presidential election (https://media.cmgdigital.com/shared/img/photos/2015/11/05/45/1e/MALL-MET-BUTTERFLY-BALLOT-2.jpg).

    • That link didn’t work, but if you follow the link in the original post, it goes over the “butterfly ballot” and includes an image. The point is really that this is just an extreme case of a problem that crops up in less extreme forms on pretty much every ballot that needs to be laid out on a 2D sheet of paper or screen.

  8. Tom Passin says:

    Not exactly about UX and ballot errors, but why should a recount be any more accurate than the original count? My rule of thumb is that human endeavors tend to have a 1 – 3% error rate at best, which seems to fit in with numbers being given here – UX problems would be expected to cause even more errors, of course.

    So I would expect a recount to also have at least that basic 1 – 3% error rate. In addition, the ballot counters would be tired by the time of the recount, and so more likely to make mistakes. If the margin of victory were less than one or two percent, the recount errors could reverse the election without having been any more accurate.

    • dhogaza says:

      Recounts rarely move results more than a tiny fraction of a percent. Some states don’t allow a recount if the spread is more than 0.5%.

      So recount history, at least recent history, in the US suggest that either ballot counting tends to be much more accurate than you assume, or that a higher error rate is systematic and repeated in the recount (which I doubt).

      There’s a lot of automation in ballot counting which probably helps account for the low error rate.

      • Anoneuoid says:

        Is there a dataset of these recount results somewhere?

      • Andrew says:

        Dhogaza,

        I think there is a higher error rate which is systematic and repeated in the recount. An example is the so-called overvote when someone checks off a candidate and also writes in his name. To throw this out is an error in the sense that it’s thwarting a voter’s intention, but if it’s the rule that it’s thrown out, it could be thrown out in the recount as well.

        The other issue is that errors can cancel out. An error will typically not be all in the direction of one candidate. Unfortunately there are situations where the error hurts one candidate more than another, as in Florida in 2000.

        • Anoneuoid says:

          Yes, we need to see a distribution of these errors along with the methodology to judge the effects.

        • dhogaza says:

          “An example is the so-called overvote when someone checks off a candidate and also writes in his name. To throw this out is an error in the sense that it’s thwarting a voter’s intention, but if it’s the rule that it’s thrown out, it could be thrown out in the recount as well.”

          I’d describe a count error as being one where the rules are met but the vote tallied incorrectly, not votes tossed because they don’t meet the rules, no matter how erroneous the rules might be. One would expect these to be tossed again on a recount.

          IIRC (and I may not be) the FL 2000 example of overvotes was made worse by poorly-written instructions (UX error)?

          Regarding errors canceling out, yes, of course …

      • If the miscounting has the same error profile as the original count or if the election is closed and the miscounting is unbiased in both cases but perhaps lower in the recount, then you’d expect them to be close percentage-wise in large N elections.

  9. Anoneuoid says:

    Where did they get that data from?

    • dhogaza says:

      Well, Wikipedia has this to say, and yes, it is not the most authoritative source but it does give three elections that were overturned:

      “Of the 4,687 statewide general elections held from 2000 to 2015, 27 were followed by a recount, and only three resulted in a change of outcome from the original count: 2004 Washington gubernatorial election, 2006 Vermont Auditor of Accounts election, and 2008 United States Senate election in Minnesota.[1]”

      Keep in mind that many states limit when recounts can be triggered. PA does not allow the loser to request a recount if the spread is 0.5% or greater. Oregon does an automatic recount if the spread is less than 0.2% and allows the loser to request a recount if it is less than 0.5%, with the loser paying the costs if the result is not overturned.

      Some states allow recounts up to 1%, I believe.

      You can look up the rules for each state if you’re motivated to do so.

  10. jim says:

    The Palm Beach county ballot is horrible!! Who would dream up a thing like that? My goodness. That’s ridiculous. Never heard of scantron I guess. Just the same, I **don’t** have a Phd and I did figure it out in less than a minute. But I guess in the booth there’s a sense of pressure. We vote by mail, so there’s plenty of time to look things over.

    All that being said, the ballots I’ve seen and used would be pretty hard fup, although I have had a ballot rejected because I forgot to sign it, which is probably the biggest reason ballots get rejected here.

    I grant it’s thwarting the voter’s intent if they mark a candidate and then write in the name and it doesn’t get counted. But how far are you supposed to do to ensure people get their intentions counted? Recently I heard of a an “oncological physio therapist” or some such thing, to help you exercise if you have cancer. Maybe we need a new professional to help us through the voting process – “enfranchisement guidance consultant”?

    We spend how much per person on education? Which in the end turns out mostly to be following instructions…soo…I guess it just gets back to my obviously outdated and – probably somewhere – disproven belief that each individual is in the best position to see to they own best interests and society shouldn’t necessarily be on the hook for liposuction.

    • jim says:

      Hot Damn!!! That’s a great idea!!!! Man, I’m ready to float my new enfranchisement firm already!

      For people who are comprehensionally challenged, we’ll offer a ballot completion service!!! We’ll conduct an interview with the enfranchisee, then use hierarchical modelling to determine how they should vote to ensure they’re voting precisely in their own best interests!! Then we’ll perform the challenging task of filling out the ballot and *guarantee that the ballot is not rejected*!!!!

      Most important of all, we’ll offer discretion: no PhD will have to worry about people knowing that they’ve had enfranchisement assistance.

      IPO, baby!

      • fogpine says:

        jim,

        You wrote,

        If people make silly mistakes like filling in a write-in and voting for a candidate for the same position, whether it’s the same name or not I don’t see why the public should suffer any expense to correct that. I forgot to sign my ballot. I didn’t blame the ballot. Everyone talks about Trump needing “big boy pants” and they’re right. People carrying the weighty issues of state on their shoulders like filling out ballots need to have big boy pants too.

        In medicine, a problem is that patients sometimes get administered the wrong dose. If you have a syringe where the “3 mL” marking is obscured by the manufacturer’s trade name, then clinicians may be especially likely to mess up when administering a dose that is supposed to be 3 mL.

        Is it syringe’s fault? Is it the clinicians’ faults? I mean, one could argue about it, but mainly just fix the design of the damn syringe! Blame is secondary.

        When dumb errors are consequential, like when patients are administered the wrong dose or in elections that depend on absurdly tight margins, then it’s worth putting in effort to get the UX right.

    • Even people who figure things out can fail to execute. Nervousness isn’t required. For example, it’s not like the keys on my keyboard move, but I still make mistakes typing. For the life of me, I’m at best 4 out of 5 on my phone keypad for my passcode. I don’t forget it, but unless I go ridiculously slowly, it just doesn’t register. And I make obvious brainos. With these butterfly ballots, you may have the whole thing figured out and think you’re following the arrow to the appropriate punch and miss it. It’s like how people can be off by one on their SAT answers if they slip up one during marking. It’s not that they don’t know how it’s supposed to be done.

      Also, not everyone has 20-20 vision. Not everyone has steady hands to fill in little circles. Not everyone has great reading comprehension. Some folks are actually dyslexic. I find as I get older, all those lessons on print size and contrast are being driven home as I struggle to read restaurant menus or the fine print on instructions. I also have a really hard time pressing the tiny buttons on a lot of the iPhone interfaces (Fitts’ law is largely ignored in mobile design). None of this is because I didn’t pay attention in school.

      I have reasonable analytic skills and reading comprehension, but the tax code is complicated enough that I don’t trust myself to figure it out. Ditto the legal system, but at least there the government recognizes that its laws are incomprehensible to a layperson, so you get a public defender appointed if you can’t afford one.

      My mom had a physical therapist while she was dying of cancer. Turns out it’s actually quite hard to exercise when you’ve had part of your lungs removed, are still recovering from a broken neck, and are disoriented from pain meds and overall mental decline with old age. Also, there’s no way she would have recovered even part way from the broken neck without incredibly good and patient PT providers, though that was before the cancer diagnosis. Those physical therapists were magicians. At the same time, most of the PT she was charged for consisted of someone poking their head in the door of her hospital room and saying “looks like she’s asleep—we wouldn’t want to disturb her” and then charging her for an hour of PT.

      • Anonymous says:

        “Even people who figure things out can fail to execute. “

        Sure. As I noted:
        1) I do think those ballots are terrible and they should be redesigned.
        2) I forgot to sign my ballot once recently and it was rejected. Shit happens.
        3) I do think that ballot is manageable, but it should be fixed, I agree.

        “Also, not everyone has 20-20 vision. Not everyone has steady hands to fill in little circles. Not everyone has great reading comprehension. “

        If people make silly mistakes like filling in a write-in and voting for a candidate for the same position, whether it’s the same name or not I don’t see why the public should suffer any expense to correct that. I forgot to sign my ballot. I didn’t blame the ballot. Everyone talks about Trump needing “big boy pants” and they’re right. People carrying the weighty issues of state on their shoulders like filling out ballots need to have big boy pants too.

        “Turns out it’s actually quite hard to exercise when you’ve had part of your lungs removed…”

        Clearly there are cases where therapy is required and reasonable, whether one is a cancer patient or otherwise.

      • Jeff says:

        +100

        Also, even if one feels no empathy for those who might run into trouble, what’s the argument against making a reasonable effort to help reduce errors? We know some things to do that cost very little.

        https://civicdesign.org/fieldguides/

        (Somewhere, I still have my set of these from backing the original Kickstarter.)

        • jim says:

          There’s no argument against making reasonable effort to ballot reduce errors and in fact despite the Palm Beach county ballots, my experience suggest that a very reasonable and extensive effort has already been made.

          “Reading comprehension” isn’t a ballot error.

        • jim says:

          “Also, even if one feels no empathy for those who might run into trouble”

          You feel “empathy” for a person who screws up their own ballot? That, my friend, is disgusting in it’s grotesque distortion what constitutes a problem deserving of empathy. I feel empathy for people in prison wrongly convicted of a crime. I feel empathy for people who are unfortunate victims of violent crime. I DO NOT feel “empathy” for someone who mismarks their ballot. Oh my goodness.

      • jim says:

        My main response will appear below eventually, I was so thrilled I forgot to put my name on it!

        But I’ve already assembled a leadership group who’s very tickled with the “enfranchisement guidance” concept, and feedback is already tweaking it for the incoming Administration to maximize per voter incomeimpact! With Biden and Harris coming, the group believes that “IPO” might not be the appropriate structure, rather – and rather more profitableumdamnkeyboardproactive – we’re looking at the “NGO”! Everyone’s thrilled with it!

    • somebody says:

      When I was a child, my parents would frequently ask for help with “computer stuff.” It all seemed pretty mundane to me. To save the document, you hit the big blue button called “SAVE.” To attach something to an email, you click the button called attach, you find the file in your folders. I told my parents anyone with a brain and eyes could figure it out, they called me a little asshole.

      Twenty years later, working in software, it still seemed to me like adults over 40 were almost uniformly complete morons. We A/B tested some new features where users could upgrade their package for a small fee by tapping on a button, then tapping again on a pop up window. The pop up window would appear, people would get confused, try and exit the pop up window, kill the app, and abandon their purchase entirely. I said maybe the feature is fine, users just need to take some more personal responsibility. If they can’t read a couple sentences to work a simple two-button UX, it’s not my job to help them. I was kicked out of the meeting, given a disciplinary warning, and was told “it actually is your job.”

      Nice to hear from you that I was right all along.

      • Anonymous says:

        > Nice to hear from you that I was right all along.

        If your job was selling something to morons and you failed, it doesn’t look like “you were right all along” :-)

        • Phil says:

          My thought as well.

          But also: FFS, if people don’t trust popup windows, don’t use a popup window. Makes me wonder whether the ‘morons’ were all on one side of that story.

          • somebody says:

            It seems I wasn’t clear; I’m being sarcastic about this idea

            > If people make silly mistakes like filling in a write-in and voting for a candidate for the same position, whether it’s the same name or not I don’t see why the public should suffer any expense to correct that. I forgot to sign my ballot. I didn’t blame the ballot. Everyone talks about Trump needing “big boy pants” and they’re right. People carrying the weighty issues of state on their shoulders like filling out ballots need to have big boy pants too.

  11. Dan F. says:

    It should be possible to implement a voting system in which each voter has a provable certificate of what he/she voted (even could correct an error prior to the vote’s being counted) and also that each vote has been counted correctly. Some exceptions would have to be made for specific technical issues, but it’s genuinely hard to understand that no movement has been made in this direction, the more so because elections in many contexts are conducted purely electronically already.

    It’s like comparing paper checks to online banking transfers. The latter may entail some errors and fraud but there’s no way it’s as easy or common as with the former.

    Not having a uniform system of personal identification is a fundamental impediment in the US.

    • Clyde Schechter says:

      “Not having a uniform system of personal identification is a fundamental impediment in the US.”

      But we do have such a system: driver licenses. Although the states have some variation in the layout of the information on the front, they all have barcoding on the back, and anywhere you go they can scan that barcode and bring up the information, no matter which state you’re from. The system does not have 100% uptake by the population, but it’s getting there. Most (all?) states will issue a non-driver ID that functions exactly like a driver license for identification purposes to people who don’t drive. Evidently, the system does not cover children too young to be eligible for a driver license in their state, and there is a gap for people who cannot afford to pay the fee, or for other reasons choose not to get one. But the vast majority of adults participate in this system, for better or for worse.

  12. Dale Lehman says:

    I do have some agreement with the need for people to take responsibility for their own actions – which includes filling out ballots correctly. But that does not excuse poor designs – the butterfly ballot being one case in point. In that particular case, it could have been ineptitude, but I suspect more sinister intentions. And, given the elderly population in Palm Beach County (including my parents, at that time, who checked each other filling out their ballots but that may not have been sufficient quality control), it should have called for even more careful design and testing than usual. Let’s not forget that their is an entire industry that thrives on subtle ways to mislead people. Take a look at https://darkpatterns.org/ to see many such examples. Even very smart people can easily be fooled into doing things that they did not intend. Ultimately they bear responsibility for their actions, but that in no way excuses people who make poor designs, even more so when it is intentional on their part.

  13. David McCormick says:

    Did you mean to link to the Caltech-MIT voting project in the article. The link is here: http://vote.caltech.edu/. The site you linked to is of course interesting and informative. Thanks.

Leave a Reply