So, I came across this news article titled, “Riley Thinks Suits Make the Coach. Research Says He Might Be Right.”:
The suit had a classic name: the Clark Gable. Navy blue and cut just right, it was the creation of Giorgio Armani, the legendary Italian designer.
It was the piece that made Pat Riley, the legendary NBA coach and executive, believe in the power of style. . . .
“I think an audience wants to see somebody on the sidelines who looks like a leader, dresses like a leader, acts like a leader,” Riley said.
It sounded like a bold claim. Sure, a business suit is undoubtedly nicer than the casual “athleisure” look — team-issue polos and pullovers — that NBA coaches adopted during the COVID-19 pandemic. But can a coat and tie really make someone more of a leader?
“It’s a perfectly reasonable thing to think,” said Abe Rutchick, a professor of psychology at California State University, Northridge. “Which is the idea that the clothes we wear have psychological meaning. We put something on, it’s not just clothes. It means something.”
Uh oh, social psychology research . . .
The article continues:
In the early 2010s, during the rise of casual attire, Rutchick and his colleagues examined a similar question and found something intriguing: Wearing formal attire might actually make a person think and act like a leader.
The researchers, using a variety of cognitive tasks, found that wearing formal clothes caused participants to shift from a concrete mode of thinking to a more abstract mindset — they thought of the big picture and looked further into the future. In other words, they thought like someone who was in charge. . . .
The paper, published in 2015, came a few years after another group of researchers found that people who wore a doctor’s white lab coat — and understood its symbolic meaning — had an increased ability to focus and pay attention. . . .
This sounds pretty bad, no joke. The early 2010s were the high-water mark of junk social psychology. This sort of study was one of the main reasons that the replication crisis became a crisis.
I thought journalists had wised up on this sort of thing, but I guess it remains afloat in the business-inspirational world of leadership.
Don’t get me wrong–I have no problem with these “leadership” stories. It’s cool to read about Pat Riley, and I have no reason to doubt that suit-wearing worked well for him. Everyone has to develop their own personal style. My problem is just with the purported scientific claims.
I found the journal article and, yeah, it’s classic replication crisis fodder:
Study 1: N = 60, p = .03
Study 2: “conceptual replication,” N = 60, p = .05 with 18 people excluded because of missing data
Study 3: N = 34, p = .02
Study 4: N = 54, p = .03 after some data were excluded
Study 5: N = 150, a mix of significant and non-significant results, conclusions made based on whether various inferences reached a significance threshold.
This is pretty much textbook bad statistical analysis of the replication-crisis variety:
– Small sample sizes and noisy data so that there’s essentially no power to detect realistic effect sizes (the kangaroo problem);
– Many researcher degrees of freedom in data exclusion, coding, and analysis, the sort of flexibility that makes it possible to achieve statistically significant p-values even in the absence of any signal;
– A bunch of p-values all in the 0.01 to 0.05 range, which is not what you’d expect from a sampling model of independent experiments (or see here);
– Flexible theories that could explain results through many sorts of interactions (the piranha problem);
– No preregistered replications.
That’s just how they did things back in 2015 so I’m not trying to single out these particular researchers. We know better now. We know not to trust this sort of claims. We don’t need to find a Wansink- or Ariely-style smoking gun; nobody’s suggesting there’s fraud here; it’s just standard-issue junk science of the sort that, until recently, was regularly published in major psychology journals and was regularly featured uncritically in major news media.
The only notable thing to me is to see this sort of claim being pushed in the New York Times now, because I had the vague impression that journalists were now aware of the replication crisis. But I guess there’s still a reservoir of credulity for such claims for stories related to the fuzzy topic of business leadership. I’d hope that straight-up sports reporting would have higher standards for the reporting of research on human performance.
P.S. This is an appropriate post for July 4th now that junk science is ensconced in the U.S. government.
“Small sample sizes and noisy data so that there’s essentially no power to detect realistic effect sizes (the kangaroo problem)”
Yes, that is a classical problem, we get it, but it is perhaps more interesting to re-frame it somewhat for empirical research in other fields. For instance, I am currently writing a paper with n = 200,000 and non-noisy (in principle, deterministic reliability-wise) data, but with no prior studies saying anything about theoretically reasonable effect sizes, or empirical benchmark effect sizes, and an unknown (or only approximately known) population size, as it almost always is the case in the domain of this paper. Of course, I will write that it is exploratory, but it would be interesting to read things going beyond these n < 500 fields.
When it comes to the notion that clothes influence an audience, there is the classic
The American Statistician Volume 49, 1995 – Issue 3
Imposing Views, Imposing Shoes: A Statistician as a Sole Model
Paul Alper
& Robert L. Raymond
https://www.tandfonline.com/doi/abs/10.1080/00031305.1995.10476172
It is now long forgotten, but back then, Bob Raymond and I were sort of famous, at least on local TV, for our Nike sneaker contract. Unlike Pat Riley, our conclusion was that wearing Nikes resulted in a less favorable response of the student evaluations.
I still have the Nike sneakers, and as yet I have not worn them. The sneakers are adjacent to my K&E slide rule.
Paul: $64 to read the article! I think that might be more expensive than the sneakers. At least I now know to wear a business suit, dress shoes, and adopt a power pose. Since I do none of those things, I guess that explains my career.
On the contrary: we already new that power poses were effective. Now we know that dressing for success works. But which is more significant? Can we adopt a power pose wearing sweatpants and achieve the same results as wearing a business suit but slouching? So many more studies to conduct, so many more papers to publish, so many more headline stories to report, and so many more careers to promote. But really – a lot of money is at stake. Think of the total dollars involved: $56 Billion on personal mentoring globally and $586 Billion on clothing in the US alone. It may be junk science, but it has an impact.
We need to start this research program at the basic level. Hypothesis: wearing clothes makes people feel more confident in social interactions. Once we establish some foundational knowledge, we can move on to refinement.
I get why you use the “who could have known, now we know better” framing. But really in 2015 people were talking about this problem for over 50 years already. Even without access to that literature, all it requires is to think for oneself rather than resort to authority/consensus heuristics.
+1
This also reminds me of what you sometimes read about that “everyone does this” and “let’s focus on the system or the incentives and not the individual people” and things in that line. What I wonder is whether that is an appropriate or fair way to look at things.
Perhaps the more critical or better people left academia, or were driven out by those that performed research sub-optimally over those 50+ years. If you are then left with a majority of scientists who perform research poorly, or are incompetent, is it appropriate or fair to say “well, everyone does it” and “let’s not focus on the individual here”?
What if you all kind of together have created a majority group of bad scientists who all kind of look the other way, or pretend things aren’t happening, or are incompetent, or are kind of working together in a way to keep doing what you’re doing, or behaving immorally?
Thanks. I can well-understand why you are called AAAnonymous.
In reality, it is practically impossible to not make honest “mistakes” in empirical sciences; after all, to me, that is the whole takeaway from the garden of forking paths.
And, in many cases, it is even impossible to say whether these really are “mistakes”. Surely you do not want to kill all creativity with rigid protocols and such? To put aside randomized clinical trials, in how many studies you can even do pre-registrations without committing worse sins (i.e., looking at the data is often a good thing)?
Therefore, I find all this ranting and raving (see the previous post) distasteful and unhelpful for the cause. That said, there are also some good points in the backlogs (such as that you shouldn’t trust a single study, etc.).
I can clearly remember my thesis in my final year of my research master’s degree in behavioural science. I had designed an experiment, and had to analyze my data. This was around 2010-2011, I believe I had not had a proper education in how to analyze data according to the now “new” and non-QRP ways. In fact, if I remember correctly I listened to someone explain what I remember was a whole lot of things that did not make sense to me concerning writing up one’s paper in a class I voluntarily took. I stopped taking that class pretty soon, as I felt it all just didn’t make much sense.
AAAnyway, we did have to sort of make a research proposal for that final year study and had to write down the hypothesis and things like that if I remember correctly. Kind of like a pre-registration in a way perhaps. I clearly remember analyzing the data in many ways to see what could be found, because that’s kind of what I thought I was supposed to do. After that, I had the thought that I should simply report what I set out to investigate at the start of the whole thing, and I did that if I remember correctly. If I can have that thought as a final year research master student around 2010-2011, I think full professors should be able to have thought that as well.
Should you be interested in that final year research master study, you can look it up, as it was published. It’s titled “Associations between psychologist’s thinking styles and accuracy on a diagnostic classification task”. Check out the samples sizes of 22 and 25 or whatever they were! I even wondered about that at the time, and whether it would be appropriate or even necessary to replicate the study with a larger sample size. As it was my final year, and due to some other reasons, that didn’t happen. Still, I wondered about it at that time.
Also check out some of my other stuff on SSRN if you are interested! (“Psychological science replicates just fine, thanks” and “Pre-registration, grocery lists, and particular pre-registration issues” might both be interesting for you as further information that may lead to the conclusion that it might be the case that I at least try to think about things).
Jukka:
My “ranting and raving” is not about people making mistakes, it’s about people refusing to learn from their mistakes once those mistakes have been pointed out.
Regarding preregistration: as I’ve written elsewhere, preregistration is a floor, not a ceiling. If you preregisters an analysis, there’s nothing stopping you from doing lots more analyses.
It is fine. After all, you’re surfing on the Internet! I was just trying to say that anonymous ranting and streams of consciousness are, well, yeah. But I had some good points too, I believe, so I hope Andrew, Jessica, and maybe others, could contribute beyond ranting in future posts.
Jukka:
Hey, we don’t just rant! We do lots of research; see here. There’s research in a lot of our blog posts too. So, yes, we’ve contributed a lot beyond ranting in past posts, and we will contribute a lot beyond ranting in future posts.
The ranting is helpful too, not just in providing comfort to young researchers who email me in frustration because their bosses are trying to get them to perform research misconduct, but also in giving us insight into important statistical ideas. For example, ranting blog posts eventually led to these two influential papers:
– [2019] Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business and Economic Statistics 37, 447-456? (Andrew Gelman and Guido Imbens)
– [2014] Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science 9, 641-651. (Andrew Gelman and John B. Carlin)
So I stand by my rants.
Perhaps it’s also useful to mention that if my specific comment(s) on this blog post here is seen as “ranting” that does not mean or imply that I haven’t commented other non-ranting things on this blog. Here are two examples, which some might even view as “constructive” or “helpful” or whatever term they might want to use. I posted the following using Anonymous as a handle, but I am the same person.
Here’s an idea about performing psychological science research (possibly compare to that retracted Protzo et al. 2023 retracted paper for some extra thoughts about certain things):
https://statmodeling.stat.columbia.edu/2017/12/17/stranger-than-fiction/#comment-628652
And here’s an idea about publishing and peer review:
https://statmodeling.stat.columbia.edu/2025/08/03/you-can-cite-peer-reviewed-research-in-support-of-almost-any-claim-no-matter-how-absurd/#comment-2401742
Hi Andrew! I’ve tried, in the media conversations I’ve had since this paper came out, to discuss the replication crisis in framing the paper. Rarely does that context make it into the article, even when the journalist understands it (which this guy did) – he’s got to frame his piece in a way that’s digestible for his audience. Hence my line that “it’s a perfectly reasonable thing to think,” which was accompanied by considerable skepticism about our actual findings. But if he prints all that, his article flops.
Agreed that this paper is apex Replication Crisis. I will say that, to my recollection, when we ran these studies (mostly in 2010 I think?), we didn’t engage in chicanery. (We even had another study in 2016 with N=200 showing formal clothing influencing delay discounting, and yet another NOT showing that effect.)
Anyway, I’m glad we’re in a better world now. And I appreciate the rants.
Abe:
Thanks for the update, and I’m happy to continue providing free rants!