The problem with those studies that claim large and consistent effects from small and irrelevant inputs

Dale Lehman writes:

You have often critiqued those headline grabbing studies such as how news about shark attacks influence voting behavior, how the time of month/color of clothing influences voting, etc. I am in total agreement with your criticisms of this “research.” Too many confounding variables, too small sample sizes, too many forking paths, poor incentives to grab headlines, etc. But one aspect of your critique I don’t understand. You have said (my paraphrasing here) that people’s political beliefs are not as superficial and fickle as these studies claim to show. I am wondering (I don’t know this research) if your prior on this is based on evidence or is it mood affiliation on your part? While I don’t find any of these studies (the ones you critique) convincing, my own prior is that people’s voting behavior is indeed fickle and superficial. The last presidential election is but a glaring example of this. Repeatedly, people seem to vote according to what seem like frivolous and easily manipulated perceptions. Is there a disconnect between your views on voters’ beliefs and your critique of research which seems to portray voters as easily manipulated? Or are you saying voters are not easily influenced but their beliefs may be based on superficial and irrational perceptions?

My reply:

First, let’s separately consider primary and general elections. Primary elections are hard to predict because the candidates have the same party affiliation and typically have similar positions, voters often don’t have much time to think about their choice, and there can be many candidates running. General elections are much more patterned.

I don’t think most vote choices in the general election are superficial or fickle. Most people vote their party ID, and we saw this in 2016 as well as 2014, 2012, 2010, etc.

I agree, however, that some people vote based on superficial or fickle reasons, and these choices can make a difference in a close election.

But the papers on ovulation and voting, shark attacks and voting, college football and voting, etc., don’t just say that voters, or some voters, are superficial and fickle. No, these papers claim that seemingly trivial or irrelevant factors have large and consistent effects, and that I don’t believe. I do believe that individual voters can be influenced these silly things, but I don’t buy the claim that these effects are predictable in that way. The problem is interactions. For example, the effect on my vote of the local college football team losing could depend crucially on whether there’s been a shark attack lately, or on what’s up with my hormones on election day. Or the effect could be positive in an election with a female candidate and negative in an election with a male candidate. Or the effect could interact with parent’s socioeconomic status, or whether your child is a boy or a girl, or the latest campaign ad, etc.

P.S. Thanks to Diana Senechal for the above photo.

2 thoughts on “The problem with those studies that claim large and consistent effects from small and irrelevant inputs

Leave a Reply to Hence Cancel reply

Your email address will not be published. Required fields are marked *