This came up in our piranha paper, and it’s convenient to have these references in one place:
Here are just some of the factors that have been published in the social priming and related literatures as having large and predictable effects on attitudes and behavior: hormones (Petersen et al., 2013; Durante et al., 2013), subliminal images (Bartels, 2014; Gelman, 2015b), the outcomes of recent football games (Healy et al., 2010; Graham et al., 2022; Fowler and Montagnes, 2015, 2022), irrelevant news events such as shark attacks (Achen and Bartels, 2002; Fowler and Hall, 2018), a chance encounter with a stranger (Sands, 2017; Gelman, 2018b), parental socioeconomic status (Petersen et al., 2013), weather (Beall and Tracy, 2014; Gelman, 2018a), the last digit of one’s age (Alter and Hershfield, 2014; Kühnea et al., 2015), the sex of a hurricane name (Jung et al., 2014; Freese, 2014), the sexes of siblings (Blanchard and Bogaert, 1996; Bogaert, 2006; Gelman and Stern, 2006), the position in which a person is sitting (Carney et al., 2010; Cesario and Johnson, 2018), and many others.
These individual studies have lots of problems (see references below to criticisms); beyond that, the piranha principle implies that it would be very difficult for many of these large and consistent effects to coexist in the wild.
References to the claims:
Kristina M. Durante, Ashley Rae, and Vladas Griskevicius. The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science, 24:1007–1016, 2013.
Larry Bartels. Here’s how a cartoon smiley face punched a big hole in democratic theory. Washington Post, https://www.washingtonpost.com/news/monkey-cage/wp/2014/09/04/heres-how-a-cartoon-smiley-face-punched-a-big-hole-in-democratic-theory/, 2014.
A. J. Healy, N. Malhotra, and C. H. Mo. Irrelevant events affect voters’ evaluations of government performance. Proceedings of the National Academy of Sciences, 107:12804–12809, 2010.
Matthew H. Graham, Gregory A. Huber, Neil Malhotra, and Cecilia Hyunjung Mo. Irrelevant events and voting behavior: Replications using principles from open science. Journal of Politics, 2022.
C. H. Achen and L. M. Bartels. Blind retrospection: Electoral responses to drought, flu, and shark attacks. Presented at the Annual Meeting of the American Political Science Association, 2002.
Anthony Fowler and Andrew B. Hall. Do shark attacks influence presidential elections? Reassessing a prominent finding on voter competence. Journal of Politics, 80:1423–1437, 2018.
Melissa L. Sands. Exposure to inequality affects support for redistribution. Proceedings of the National Academy of Sciences, 114:663–668, 2017.
Michael Bang Petersen, Daniel Sznycer, Aaron Sell, Leda Cosmides, and John Tooby. The ancestral logic of politics: Upper-body strength regulates men’s assertion of self-interest over economic redistribution. Psychological Science, 24:1098–1103, 2013.
Alec T. Beall and Jessica L. Tracy. The impact of weather on women’s tendency to wear red or pink when at high risk for conception. PLoS One, 9:e88852, 2014.
A. L. Alter and H. E. Hershfield. People search for meaning when they approach a new decade in chronological age. Proceedings of the National Academy of Sciences, 111:17066–17070, 2014.
Kiju Jung, Sharon Shavitt, Madhu Viswanathan, and Joseph M. Hilbe. Female hurricanes are deadlier than male hurricanes. Proceedings of the National Academy of Sciences, 111:8782–8787, 2014.
R. Blanchard and A. F. Bogaert. Homosexuality in men and number of older brothers. American Journal of Psychiatry, 153:27–31, 1996.
A. F. Bogaert. Biological versus nonbiological older brothers and men’s sexual orientation. Proceedings of the National Academy of Sciences, 103:10771–10774, 2006.
D. R. Carney, A. J. C. Cuddy, and A. J. Yap. Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21:1363–1368, 2010.
References to some criticisms:
Andrew Gelman. The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management, 41:632–643, 2015a.
Andrew Gelman. Disagreements about the strength of evidence. Chance, 28:55–59, 2015b.
Anthony Fowler and B. Pablo Montagnes. College football, elections, and false-positive results in observational research. Proceedings of the National Academy of Sciences, 112:13800–13804, 2015.
Anthony Fowler and B. Pablo Montagnes. Distinguishing between false positives and genuine results: The case of irrelevant events and elections. Journal of Politics, 2022.
Andrew Gelman. Some experiments are just too noisy to tell us much of anything at all: Political science edition. Statistical Modeling, Causal Inference, and Social Science, https://statmodeling.stat.columbia.edu/2018/05/29/exposure-forking-paths-affects-support-publication/, 2018b.
Andrew Gelman. Another one of those “Psychological Science” papers (this time on biceps size and political attitudes among college students). Statistical Modeling, Causal Inference, and Social Science, https://statmodeling.stat.columbia.edu/2013/05/29/another-one-of-those-psychological-science-papers/
Andrew Gelman. When you believe in things that you don’t understand. Statistical Modeling, Causal Inference, and Social Science, https://statmodeling.stat.columbia.edu/2014/04/15/believe-things-dont-understand/, 2018a.
Simon Kühnea, Thorsten Schneiderb, and David Richter. Big changes before big birthdays? Panel data provide no evidence of end-of-decade crises. Proceedings of the National Academy of Sciences, 112:E1170, 2015.
Jeremy Freese. The hurricane name people strike back! Scatterplot, https://scatter.wordpress.com/2014/06/16/the-hurricane-name-people-strike-back/, 2014.
Andrew Gelman and Hal Stern. The difference between “significant” and “not significant” is not itself statistically significant. American Statistician, 60:328–331, 2006.
J. Cesario and D. J. Johnson. Power poseur: Bodily expansiveness does not matter in dyadic interactions. Social Psychological and Personality Science, 9:781–789, 2018.
Lots more out there:
The above is not intended to be an exhaustive or representative list or even a full list of examples we’ve covered here on the blog! There’s the “lucky golf ball” study, the case of the missing shredder, pizzagate, . . . we could go on forever. The past twenty years have featured many published and publicized claims about essentially irrelevant stimuli having large and predictable effects, along with quite a bit of criticism and refutation of these claims. The above is only a very partial list, just a paragraph giving a small sense of the wide variety of stimuli that are supposed to have been demonstrated to have large and consistent effects, and it’s relevant to our general point that it’s not possible for all these effects to coexist in the world. Again, take a look at the piranha paper for further discussion of this point.
I am skeptical about many of these studies of subtle influences – and by the fact that none of the people involved had anything to say about subtle influences from masking entire populations – but there are simple physical mechanisms which show some Piranha effects. I have two light switches, at the top and bottom of a flight of stairs, controlling a single light bulb. This can be generalised to any desired number of switches. One way to do this is to have a pair of power wires, of which only one is live at any point, and a single return wire. Each switch either swaps or does not swap the connections between the incoming and outgoing pair of power wires, depending on its position, so moving any single switch will turn the light off it it was on, or on if it was off (https://en.wikipedia.org/wiki/Multiway_switching#More_than_two_locations).
If you run an experiment in which you vary one switch while holding the other switches constant, you will find that the switch you vary accounts for 100% of the variance in light output, no matter which switch you choose.
Quote from above: “I am skeptical about many of these studies of subtle influences – and by the fact that none of the people involved had anything to say about subtle influences from masking entire populations – but there are simple physical mechanisms which show some Piranha effects.”
I have recently began to view certain, if not many, effects and constructs and “theories” in social science research as part of some imaginary experiment.
I think it’s pretty interesting to think about what being exposed to countless of these “studies of subtle influences” and other “findings” reported on in the papers over the years might do to people reading that stuff.
Do they start to believe that “subtle influences” lurking around each corner affect their decisions and actions? Do they develop some sort of “dependency” on their “environment” and “the system” that is apparently responsible for these “subtle influences”? Do they begin to think they really don’t have any choice in things, or free will, or stuff like that?
You could view certain, if not many, things in social science from that perspective, if only in light of some possibly interesting and amusing thought experiment.
I was thinking about the (in-)famous remark by Kahneman about these kinds of subtle effects that goes something like “You have not choice but to accept these findings.”. When typing something like that in my searchbar, I came across the following post on this blog that also mentions the (in-)famous remark by Kahneman:
https://statmodeling.stat.columbia.edu/2014/09/03/disagree-turing/
“The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true.”
Anon:
Yes, Kahneman has retracted that particular statement. Unfortunately, Turing did not live long enough to reassess his comparable statement about the evidence for extra-sensory perception.
Kahneman has commented on this in a critique of the priming chapter in his book.
https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/
This reminds me of this comment by Kahneman.
I accept the basic conclusions of this blog. To be clear, I do so (1) without expressing an opinion about the statistical techniques it employed and (2) without stating an opinion about the validity and replicability of the individual studies I cited.
What the blog gets absolutely right is that I placed too much faith in underpowered studies. As pointed out in the blog, and earlier by Andrew Gelman, there is a special irony in my mistake because the first paper that Amos Tversky and I published was about the belief in the “law of small numbers,” which allows researchers to trust the results of underpowered studies with unreasonably small samples. We also cited Overall (1969) for showing “that the prevalence of studies deficient in statistical power is not only wasteful but actually pernicious: it results in a large proportion of invalid rejections of the null hypothesis among published results.” Our article was written in 1969 and published in 1971, but I failed to internalize its message.
My position when I wrote “Thinking, Fast and Slow” was that if a large body of evidence published in reputable journals supports an initially implausible conclusion, then scientific norms require us to believe that conclusion. Implausibility is not sufficient to justify disbelief, and belief in well-supported scientific conclusions is not optional. This position still seems reasonable to me – it is why I think people should believe in climate change. But the argument only holds when all relevant results are published.
I knew, of course, that the results of priming studies were based on small samples, that the effect sizes were perhaps implausibly large, and that no single study was conclusive on its own. What impressed me was the unanimity and coherence of the results reported by many laboratories. I concluded that priming effects are easy for skilled experimenters to induce, and that they are robust. However, I now understand that my reasoning was flawed and that I should have known better. Unanimity of underpowered studies provides compelling evidence for the existence of a severe file-drawer problem (and/or p-hacking). The argument is inescapable: Studies that are underpowered for the detection of plausible effects must occasionally return non-significant results even when the research hypothesis is true – the absence of these results is evidence that something is amiss in the published record. Furthermore, the existence of a substantial file-drawer effect undermines the two main tools that psychologists use to accumulate evidence for a broad hypotheses: meta-analysis and conceptual replication. Clearly, the experimental evidence for the ideas I presented in that chapter was significantly weaker than I believed when I wrote it. This was simply an error: I knew all I needed to know to moderate my enthusiasm for the surprising and elegant findings that I cited, but I did not think it through. When questions were later raised about the robustness of priming results I hoped that the authors of this research would rally to bolster their case by stronger evidence, but this did not happen.
I still believe that actions can be primed, sometimes even by stimuli of which the person is unaware. There is adequate evidence for all the building blocks: semantic priming, significant processing of stimuli that are not consciously perceived, and ideo-motor activation. I see no reason to draw a sharp line between the priming of thoughts and the priming of actions. A case can therefore be made for priming on this indirect evidence. But I have changed my views about the size of behavioral priming effects – they cannot be as large and as robust as my chapter suggested.
I am still attached to every study that I cited, and have not unbelieved them, to use Daniel Gilbert’s phrase. I would be happy to see each of them replicated in a large sample. The lesson I have learned, however, is that authors who review a field should be wary of using memorable results of underpowered studies as evidence for their claims.
https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/
Ali’s quote from Kahneman:
That’s surely wrong but maybe highlights a difference between social science and “hard” science approaches to evidence. A “conclusion” is not evidence – it’s an interpretation of observations and the observations might be fine without the conclusions necessarily being correct. That is particularly the case for “implausible” conclusions which Kahneman emphasises here. His example of climate change is an odd one since there’s nothing implausible about climate change (evidence and mechanistic understanding are aligned).
A lot of these discussions of science boil down to how we act in the face of evidence. If we’re interested in the progresson of a scientific field or in “finding “stuff out”, then as scientists we don’t have to “believe that conclusion” even if it’s supported by a large body of evidence. We’re very likely to take note of it, but perhaps we feel that the evidence is consistent with an alternative conclusion and in any case we’re aware of the provisional nature of scientific conclusions.
Actually the “Readiness Potential” (RP) study described in comments below where Libet et al provided evidence for measurable neural correlates of decision processes happening hundreds of milliseconds before their subjects reported making the conscious decisions, is a good example. These RP’s were widely reported and have been interpreted to provide evidence against “free will”. Kahneman would say (if we take his assertions at face value) that “scientific norms” compels us that belief of this conclusion is “not optional”. However it most certainly is optional and in fact later observations are consistent with alternative conclusions.
Quote from above: “Do they develop some sort of “dependency” on their “environment” and “the system” that is apparently responsible for these “subtle influences”? Do they begin to think they really don’t have any choice in things, or free will, or stuff like that?”
After hearing all the talk about “the system” and “the incentives” in Psychological Science in relation to the many possibly problematic issues in this field, and an (in my view at least) almost total lack of talk about individual responsbility and choices and such things, in the last decade or so I think it might be appropriate to state that psychological scientists themselves may have started to think about all these things in a certain way…
I would like to quote the following sentence from the blogpost in 2014 referred to above in this light: “Ahhhh, but we do have a choice!”
Regarding free will, this recent book argues that little or none exists: https://www.amazon.com/Determined-Science-Life-without-Free/dp/0525560971/ref=asc_df_0525560971/?tag=hyprod-20&linkCode=df0&hvadid=642118117378&hvpos=&hvnetw=g&hvrand=3190398084991469514&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9019526&hvtargid=pla-1945830190930&psc=1&mcid=331f84fc4ca93aef8ebf2c7916342bdf. I’m not saying I agree, but I admit it is possible that everything is predetermined and only our ignorance accounts for it not seeming so. Even if you buy his argument, I don’t think that is necessary or sufficient to lend credence to these subtle influence studies – but it is not a hard leap to go from no free will to believing that many such influences exist. In both cases, I think it is hard to disprove such views though I find it hard to believe them.
What I find interesting is how easy it is to reject subtle influences because they easily could lead you to the conclusion that there is no free will. Where is the proof that free will exists?
I subscribe to a more practical philosophy – I don’t know how to proceed if there was no free will. I also don’t know how to act if all of these subtle influences really exist simultaneously. But that’s about as deep as I can get into such philosophical questions.
Quote from above: “I subscribe to a more practical philosophy (…)”
I am very practical as well, and I don’t really care for too deep thinking or too complicated views about things.
I can clearly remember it being mentioned during my time at university in a class that there was a certain paper that measured neurons firing in the brain or something like before an action or choice was performed or mentioned, which was interpreted as being in line with there being no free will or someting like that.
I am possibly totally misrepresening the paper and findings and conclusions, but I was reminded of that when viewing the information about the recent book you link to.
My response to the teacher talking about this stuff in class was mentioning that perhaps deciding NOT to engage in something, or NOT following impulses which may arise automatically due to “subtle influences” might also be seen as an example of having free will. But, I don’t know what is exaclty seen as free will in all of this.
Anonymous: You’re probably referring to the work of Benjamin Libet, particularly the “clock-watching” experiments he undertook in the 80s: People would watch a dot rotating around a circle, and were instructed to make a “spontaneous” movement and report the position of the dot at the time when they had decided to move. Libet showed that the pre-movement brain activity substantially preceded the reported decision times. The work has been heavily criticised on both methodological grounds and in terms of its implications, but I don’t think the finding itself is overly surprising, given that only a small fraction of neural activity seems to make it to conscious awareness.
What does surprise me is that people would be shocked that our choices are a result of an interaction between our genes and our environments. I mean… what else is there? Some sort of dualist arrangement where our bodies are just puppets of a disembodied soul?
Yes, I think I remember that name in relation to the example.
Nice one!
+1
” Where is the proof that free will exists? ”
Common sense.
On Sept 17, I bought a scale and started trying to lose weight. I have since lost 40lbs. I chose to do that. There’s no other sensible conclusion, to believe otherwise *without evidence that my choice was predetermined* is contrary to simple human reason.
People have senses. Those senses work.
chipmunk
You fail at logic. Common sense? You think you chose but it is entirely possible that your “choice” was predetermined by genetics, environment, divine intervention,…. I’m not saying I believe these things, but you can’t disprove them nor can they be proven. As a practical matter – and to keep my own sanity – I also believe in free will. But that doesn’t mean that an appeal to “common sense” is proof that free will exists. To go full circle, I guess you are saying that you choose to believe that free will exists.
It depends on what you mean by “predetermined”, but of your contributors, “genetics, environment, divine intervention”, only “divine intervention” could be said to be a predeterminer of (as opposed to an “influence” on) our choices. Obviously our choices are constrained by our genetics and environment but that’s separate from the question whether we have “free will”.
So what do you mean by “predetermined”? Is it “predetermined” in the sense that given initial conditions, e.g. on Earth 4.6 billion years ago, that every particle subsequently followed the laws of physics in its interactions such that the evolution of the Earth through the subsequent billions of years (including Chipmunk’s weight loss 4.6 billion years later) was entirely defined by the initial conditions? Or are you using a different meaning of “predetermined”?
I am using “predetermined” in the sense that the combination of genetics, environment, history, etc. made our “choices” inevitable. It is only our ignorance of all these factors that makes it seem as if we actually had a choice. Note that I am not defending this position – indeed, I find it unsatisfactory. But that doesn’t mean I can disprove it.
Ag:
Regarding the light switch example . . . If I’m understanding this correctly, then yes, flipping any switch from the up to the down position (for example) has a large effect, but it’s a highly variable effect that depends on the positions of the other switches. Flipping switch A could turn the light form on to off, or from off to on. This represents a very large interaction effect that depends on other factors in the environment (in this case, the positions of the other switches). This is all consistent with the piranha principle, in that it would not be possible for all the switches to have large and consistent effects in this scenario.
I see – thanks for that
Quick comment, the first paper you mention on “male and female hormones (Petersen et al., 2013..)” has nothing to do with hormones.
Harry:
Yes, good point. That paper talks about fat arms, not hormones. I’ll rewrite the above post to fix that.
For most these, okay, sure. But the idea that the “outcomes of recent football games” have “large and predictable effects on attitudes and behavior,” is not controversial. It’s fairly undisputable depending on how you define “effects on attitudes and behavior.” For example, the city of Philadelphia makes pretty reliable predictions on the effects of the outcome of football games. It’s why they have to stockpile lightpole grease in anticipation of victories.
Here’s another citation for the list on that topic:
Hannah S. Lacqueur & Ryan W. Copus, Entertainment As Crime Prevention: Evidence from Chicago Sports
Games., 20 Journal of Sports Economics 344 (2018)
Games are a public spectacle. It’s not unreasonable to believe that there are strong connections between sports and political attitudes. After all, one of the most infamous event in the history of the Byzantine Empire was the Nika Riots which occurred when partisanship for chariot racing teams became wrapped up with support or antagonism for a whole host of policies. The riots started at the Hippodrome.
Dalton:
Yes, I agree that outcomes of football games will affect behavior. As will shark attacks! The problem is with the claim that these will have large and consistent effects. I think it makes more sense to expect small effects most of the time, with some special cases where there are large effects that sometimes are positive and sometimes are negative, hence very hard to detect with the usual noisy experiments attempting to estimate average treatment effects.
Specifically regarding, Healy et al. and the Fowler follow-up, I don’t think that Healy’s hypotheses is all that unreasonable. Sports are public spectacle and they have effects on peoples’ moods. Victories and defeats can be memorable, particularly for bigger games. For example, there are large numbers of people in Philadelphia that vividly remember the Eagles Super Bowl victory. Maybe some of those people would be more likely to vote for Joe Biden if they know that Jill Biden is an Eagles supporter? If we allow that people attach all kinds of attitudes to the incumbent in elections, even if in reality the incumbent bears little responsibility and has little influence on the factors affecting those attitudes (e.g. the current rate of inflation is a worldwide phenomenon, not the fault of a particular incumbent); then doesn’t it seem at least plausible that more clearly irrelevant things can be attached to the incumbent.
So I at least am willing to accept Healy’s hypothesis. I guess my question for you is: is the issue that Healy’s hypothesis is ridiculous or is it that the methods to test the hypothesis are inadequate? If it’s the latter, is there anyway to effectively test a hypothesis that sporting event outcomes (or other large cultural events) can affect voting patterns?
Dalton:
Again, I can buy the occasional large effect, including possibly Philadelphia in 2020. What I don’t buy are the claims of large and consistent effects that were made in the literature. Just for example, you gave an example of a potentially large effect in the direction opposite to the incumbent.
There shouldn’t be any need to “buy” or “not buy” any of this garbage. Just replicated with sound analytical methods. It disappears instantly.
In the rest of science, people first test the methods to demonstrate their credibility, then they apply them to best cases, then if they are successful in several – or even several dozen – best cases they apply them to less ideal cases until ultimately the method has been demonstrated succesful dozens of times.
In social science, OTOH, people use a method that’s known to routinely fail, apply it to a highly unlikely hypothesis, then declare a great discovery when it fails. Social science is littered with grossly incompetent “researchers.” To call the “scientists” is an insult to the *actual* scientists who proceed to generate *actual* knowlege the hard way: with real intelligence and experiments.
Frankly Andrew I can’t use the words on this blog that should be used for these people.
Quote 1 from above “Social science is littered with grossly incompetent “researchers.””
There is a paper titled “The natural selection of bad science” and I used that title as a starting point concerning the writing of a manuscript titled “The natural selection of bad psychological scientists”. With all due respect to those social scientists with good intentions, and perhaps even good capabilities, etc. I think there should be more talk about the intentions and capabilities of certain “scientists”. And there should be more talk about the personalities of these “scientists” I think as well.
Quote 2 from above: “Frankly Andrew I can’t use the words on this blog that should be used for these people.”
If I remember correctly, there was a paper published a few years ago that mentioned that certain people called those who used data-sets gathered by other researchers “research parasites” or something like that. In one of my other manuscripts, I listed several problematic issues and behaviors that researchers may have engaged in, before wondering whether engaging in these things, and hereby wasting other people’s money and throwing aside scientific values, principles, and responsibilities for personal gain, might privide a more scientifically accurate description of the term “research parasite”.
I don’t think it’s particularly fun or easy to write things like that, and I often wonder if it’s going too far, but I think it might be very necessary to sometimes state things harshly. I hope it’s in line with what someone on this blog replied below one of my comments somewhere: “Harsh, but fair”.