Cherry-picking during pumpkin-picking season? (the effects of the Jan. 6th hearings)

This post is by Jeff Lax.

Is something off in the recent Talking Points Memo by Mindy Finn/Citizen Data, on the effects of the Jan. 6th hearings?  It’s about the change in people’s attitudes in a pair of surveys before and during the hearings. It is framed around the basic health of our democracy–what could be more serious? I was going to use it to respond to a student question… but then I looked closer.

https://talkingpointsmemo.com/cafe/the-jan-6-committee-is-having-a-measurable-impact-on-voter-attitudes

They basically claim only good news for democracy and with great confidence: “While there have been conflicting reports on the impact of the Jan. 6 hearings, our polling has been more conclusive. Since the hearings began, more Americans have come to view Jan. 6 as a violent attempt to overthrow the government and more Americans now see the committee’s findings as legitimate. As we look ahead to the midterms and on to 2024, I believe the committee’s communications offer a playbook to replicate on the journey to protect our democracy and thwart those who threaten it.”  They also claim that, after the Jan. 6 Committee’s hearings, to nearly quote, Americans are more likely to hold Trump accountable, that those supporting him will face headwinds, and that efforts to rebuild democracy are paying off. 

My concerns are NOT about tricky stats or surveying issues. Rather, I think they misread their own results and neglect to note contrary evidence from their own numbers. Why is the piece doing so much cherry-picking in reporting results? Why is there so much cheerleading given the number of findings that are bad or mixed? Why isn’t it at least clear on which numbers are being compared? One worries the findings are just noise and the reporting too selective to be considered properly objective.

Start with the claims that comparisons between April and July show good news: “Skeptical Americans,” Finn writes, “including those who initially believed the 2020 election was tainted through widespread voter fraud, might be changing their minds” and “the share who believed Joe Biden won the 2020 election increased by 5%” and “Nearly twice the number of Americans who view the 2020 election as ‘stolen’ and Jan. 6th as peaceful now view the events as a violent attempt to overthrow the government.” You can get to some of the underlying numbers through a link in the original post: https://citizendata.docsend.com/view/umfmknfvebgf5uru

Their analysis of the ‘Did Biden win” question seems right in isolation. Comparing July to April, more people said Biden won, fewer had doubts about this, and fewer said he did not win. But the results from that question are undercut by the numbers in the main question on Jan. 6th.

This is obscured by the non-standard (or at least overly complicated) set of options respondents are given in the key question. To answer it, you have to implicitly make three decisions with two options each, picking one of a permitted six combinations for labeling Jan. 6th: (1) “Violent attempt to overthrow the government in response to a legitimate election,” (2) “Peaceful protest against a stolen election,” (3) “Violent protest against a legitimate election,” (4) “Violent attempt to overthrow the government in response to a stolen election,” (5) “Violent protest against a stolen election,” or (6) “Peaceful protest against a legitimate election.”  (For better or worse, they don’t allow all eight possibilities, so you can’t say “peaceful attempt to overthrow the government in response to a legitimate election” or same “… against a stolen election.”)

So if you think Jan. 6th was “violent,” you narrow it down to options 1, 3, 4, or 5. But then do you think it is “attempt to overthrow” (1 or 4)? Or “protest” (3 or 5)? Then you still have to choose whether the election was “legitimate” (1 or 3) or “stolen” (4 or 5). If you start with “peaceful,” you then have to contend with two options, “stolen” (2) or “legitimate” (4), distributed among the six options in total. Or you can start with “legitimate election” (1, 3, or 6) or “stolen” (2, 4, and 5). Then you have to do “violent attempt” or “violent protest” or “peaceful protest”. So messy. So hard on the survey respondent. Too hard? Perhaps even too hard to get the write-up right?

Here is what the results look like, tabulated from the “behind-the-scenes” graph they link to (with less information in the article itself):

category april july change
violent attempt to overthrow the government in response to a legitimate election 1 34.1 33.9 -0.2
peaceful protest against a stolen election 2 16.1 14.6 -1.5
violent protest against a legitimate election 3 12.4 13.7 1.3
violent attempt to overthrow the government in response to a stolen election 4 6.5 11 4.5
violent protest against a stolen election 5 9.6 9.6 0
peaceful protest against a legitimate election 6 3.7 6.1 2.4

Do we see only good news? If you think (1) (‘violent-overthrow-legitimate’) is correct, then it would be good news if that percentage went up. It didn’t (-.2% April to July). True, fewer (-1.5% April to July) falsely think it a (2) ‘peaceful-protest-stolen’ (that’s good), but more (+2.4%) think it only a (6) ‘peaceful-protest-legitimate’ (that’s bad, or mixed). Those who said (3) ‘violent-protest-legitimate’ went up (+1.3%, that’s bad, or mixed). There was no change in (5) at all (+0.0%). Those who chose 6, just ‘peaceful-protest-legitimate’, went up by (+2.4%, that’s bad, or mixed).

Where did the main claim come from, that nearly twice the number who said ‘peaceful-stolen’ in April say ‘violent-overthrow-legitimate’ in July? Maybe the write-up was meant to rephrase this text in the linked report: “At the conclusion of the hearings in July, the percentage of Americans who viewed January 6th as a “violent attempt to overthrow the government in response to a stolen election” nearly doubled from 6.5% to 11%; most of this increase came from those who had previously indicated they were uncertain about the events of January 6th.” That’s category 4’s pair of numbers. But that’s not the same as the main text’s “Nearly twice the number of Americans who view the 2020 election as ‘stolen’ and Jan. 6 as peaceful now view the events as a violent attempt to overthrow the government.”  (Also, any cross-tabs showing where “most of this increase came from” are not shown.)  

I suppose they could be comparing category 1 in July at 33.9% (violent-overthrow-legitimate) to category 2 in April at 16.1% (peaceful-protest-stolen) but the former is MORE than twice the latter. And I don’t think that comparison is meaningful really.  Heck, one could also say more than twice the number who said that January 6 was ‘violent-overthrow-stolen’ (6.5%, category 4, April) later say it was a ‘peaceful protest against a stolen election’ (14.6%, category 2, July). How dramatically bad does that sound! Moreover, one could have said using only April to April comparisons that over twice as many said good category 1 as bad category 2.  Again, I’m not sure which results are being discussed in the main finding or why it makes sense to arbitrarily compare categories and time periods with so many comparisons that could be done.

What can we safely say?  Those who admit Jan. 6th was “violent” did go from 62.6 to 68.2 (summing categories, an increase of 5.6%). So that’s good news. Yet a quarter of that overall increase is the increase in the people who said it was only a ‘violent protest’ of a ‘legitimate election’ (there was no change in those who said ‘violent protest of stolen election’). Minimizing it as ‘protest’ (compared to ‘attempt to overthrow’) doesn’t seem so great. And there was an increase of .9% in those who said it was a ‘peaceful protest’ of any type (that’s bad). The percent who thought the’ election legitimate’ did increase by 3.5% (that’s good), but those who said ‘stolen’ went up nearly as much, 3% (that’s bad).  Movement seems to be coming from “don’t know” (down 6.5% from April to July), as much as from converting those with false views.

Turning to the question on the legitimacy of the committee on the 6th, more say ‘legitimate’ and their recommendations should be ‘seriously considered’ (good!), but more say ‘not legitimate’ and should be ‘ignored’ (bad!). More say legitimate but should be ‘not seriously considered.’ That’s… bad? Good? No idea. And how is ‘not seriously considered’ different from ‘ignored’? I give up.

 

8 thoughts on “Cherry-picking during pumpkin-picking season? (the effects of the Jan. 6th hearings)

  1. Jeff:

    Setting aside the thing about the survey question, let me just emphasize that, historically, fascists typically seize power through some mixture of public opinion, legal violence, and illegal violence. So it’s not like they need 50% support.

    • This is a good point, but it’s also worth noting that Hitler and Mussolini and Dollfus, and more recent people who would probably object to being called fascists but whom I think many people would agree are at least “fascim-adjacent” ( Orban, Bolsonaro, Trump, Ergodan), were elected by majorities or near-majorities. At least I think that’s the case, actually I’m not sure how the electoral system works in all of these places (and yes of course I know Trump didn’t get over 50% of the vote, that’s why I said ‘near-majorities’). There might also be fascists who managed to take power in a democracy without a lot of popular support, but I can’t think of any.

      • Phil:

        No, Hitler was not elected by a majority or near-majority.

        The last free election of that period was the legislative election in 1932, when the Nazis received 33% of the vote. Earlier that year was a presidential election where Hitler got 37% of the vote. That was 37% more than he should have received, but nothing close to a majority.

        And that’s really the point of my comment: Yes, the Nazis needed a lot of popular support to get to where they were, but they didn’t need a majority; they just needed enough to be able to take power using non-democratic means.

        • “No, Hitler was not elected by a majority or near-majority.”

          Well maybe not but didn’t the Communists support some Nazi candidates in hopes that, if the Nazis won the primaries, they’d get beaten by moderates the general election? That’s a whole buncha clever right there! Makes you wonder why they weren’t running the country to begin with.

      • Trump was elected by 27% of the eligible voters, and Orban by 34% (which gave his party a super majority in parliament, due to Hungary’s peculiar system).
        Of course neither are near majorities. It is really the non-voters who are the decisive block in these cases.

  2. Other polls find pretty much no effects or partisan effects – a much more careful and forthright Monmouth analysis for example:
    https://www.monmouth.edu/polling-institute/reports/monmouthpoll_us_080922/
    See results here:
    https://www.monmouth.edu/polling-institute/wp-content/uploads/sites/22/2022/08/Jan6-opinion-1.jpg
    Was a legitimate protest went up one point. Biden voter fraud no change. Was a riot down a point. Etc.

    Also from a CNN poll “Although there was little movement on each of these questions, partisans appear to be moving in opposite directions.”
    https://www.cnn.com/2022/07/26/politics/cnn-poll-january-6-trump

    The contrast to how findings are created and presented in the Citizen Data analysis is pretty stark. CD seems to think the “conflicting” findings of other survey work must be a flaw rather than an accurate aggregate finding of some noise mixed with no effects or partisan effects that cancel out. To call their own results “conclusive,” especially but not only when they are anything but, is a rather shocking departure from objective pollster or analysis practice, no?

  3. The table had me very confused until I noticed the numbers don’t sum to 100%. I’m guessing the rest are people who answered the survey but declined to answer that particular question.

    That group shrank by 6.5 percentage points — the biggest change.

    As for legitimacy, +3.5% for believing legitimate while +3% for stolen. These are net numbers, so they don’t say whether anyone who had an opinion in April changed their minds. I guess it’s good news for people who liked the election outcome, but not very good news.

Leave a Reply

Your email address will not be published. Required fields are marked *