This is Jessica. How to weigh metascience or statistical reform proposals has been on my mind more than usual lately as a result of looking into and blogging about the Protzko et al. paper on rigor-enhancing practices. Seems it’s also been on Andrew’s mind.
Like Andrew, I have been feeling “bothered by the focus on procedural/statistical ‘rigor-enhancing practices’ of ‘confirmatory tests, large sample sizes, preregistration, and methodological transparency’” because I suspect researchers are taking it to heart that these steps will be enough to help them produce highly informative experiments.
Yesterday I started thinking about it via an analogy to writing. I heard once that if you’re trying to help someone become a better writer, you should point out no more than three classes of things they’re doing wrong at one time, because too much new information can be self-defeating.
Imagine you’re a writing consultant and people bring you their writing and you advise them on how to make it better. Keeping in mind the need to not overwhelm, initially maybe you focus on the simple things that won’t make them a great writer, but are easy to teach. “You’re doing comma splices, your transitions between paragraphs suck, you’re doing citation wrong.” You talk up how important these things are to get right to get them motivated, and you bite your tongue when it comes to all the other stuff they need help with to avoid discouraging them.
Say the person goes away and fixes the three things, then comes back to you with some new version of what they’ve written. What do you do now? Naturally, you give them three more things. Over time as this process repeats, you eventually get to the most nuanced stuff that is harder to get right but ultimately more important to their success as a writer.
But for this approach to work presupposes that your audience either intrinsically cares enough about improving to keep coming back, or that they have some outside reason they must keep coming back, like maybe they are a Ph.D. student and their advisor is forcing their hand. What if you can never be sure when someone walks in the door that they will come back a second time after you give your advice? In fact, what if the people who really need your help with their writing are bad writers because they fixated on the superficial advice they got in middle school or high school that boils good writing down to a formula, and considered themselves done? And now they’re looking for the next quick fix, so they can go back to focusing on whatever they are actually interested in and treating writing as a necessary evil?
Probably they will latch onto the next three heuristics you give them and consider themselves done. So if we suspect the people we are advising will be looking for easy answers, it seems unlikely that we are going to get them there using the approach above where we give them three simple things and we talk up the power of these things to make them good writers. Yet some would say this is what mainstream open science is doing, by giving people simple procedural reforms (just preregister, just use big samples, etc) and talking up how they help eliminate replication problems.
I like writing as an analogy for doing experimental social science because both are a kind of wicked problem where there are many possible solutions, and the criteria for selecting between them are nuanced. There are simple procedural things that are easier to point out, like the comma splices or lacking transitions between paragraphs in writing, or not having a big enough sample or invalidating your test by choosing it posthoc in experimental science. But avoiding mistakes at this level is not going to make you a good writer, just like enacting simple procedural heuristics are not going to make you a good experimentalist or modeler. For that you need to adopt an orientation that acknowledges the inherent difficulty of the task and prompts you to take a more holistic approach
Figuring out how to encourage that is obviously not easy. But one reason that starting with the simple procedural stuff (or broadly applicable stuff, as Nosek implies the “rigor-enhancing practices” are), seems insufficient to me is that I don’t necessarily think there’s a clear pathway from the simple formulaic stuff to the deeper stuff, like the connection between your theory and what you are measuring and how you are measuring it and how you specify and select among competing models. I actually think things make more sense to go the opposite way, from why inference from experimental data is necessarily very hard as a result of model misspecification, effect heterogeneity, measurement error etc. to the ingredients that have to be in place for us to even have a chance, like sufficient sample size and valid confirmatory tests. The problem is that one can understand the concepts of preregistration or sufficient sample size while still having a relatively simple mental model of effects as real or fake and questionable research practices as the main source of issues.
In my own experience, the more I’ve thought about statistical inference from experiments over the years, the more seriously I take heterogeneity and underspecification/misspecification, to the point that I’ve largely given up doing experimental work. This is an extreme outcome of course, but I think we should expect that the more one recognizes how hard the job really is, the less likely one is to firehose the literature in one’s field with a bunch of careless dead-in-the-water style studies. As work by Berna Devezer and colleagues has pointed out, open science proposals are often subject to the same kinds of problems such as overconfident claims and reliance on heuristics that contributed to the replication crisis in the first place. This solution-ism (a mindset I’m all too familiar with as a computer scientist) can be counterproductive.
For me, a better example is golf. There are at least dozens (more likely hundreds) of things involved in a golf swing from foot position, hand grip, to swing plane, to club selection, to swing speed, to lower and upper body position,….. There are many sources that tell you “the three things” that will make the biggest difference in your performance. Almost all of these “three thing advice columns” list a different 3 things. The real problem is that most of these things are related – you can make up for one deficiency by adapting some other feature: more commonly, you can get many things right, only to fail due to something else on the list not quite being right. And, unfortunately, there is no such thing as a slightly off swing – being slightly off is usually a disaster (at least for me). Further, there is no substitute for practice (except for the gifted few).
Actually, data analysis is easier. There are many things that can be a little off without causing bad problems. Some things, like heterogeneity and poor measurement and misspecification cannot be compensated for by having more data or using different algorithms. So, there is more hope for data analysis than for my golf game – admittedly a low bar to set.
I don’t golf, but I’ve flipped through my dad’s golf magazines while at my parent’s house and definitely get the ‘these three things’ vibe.
One thing that’s different about golf though is that most people who do it presumably want to. Whereas, at least in academia, there are a lot of people who treat writing as a necessary evil, who would probably never seek it out if it wasn’t required. Maybe designing and analyzing experiments is somewhere in the middle – if you really hate it you won’t end up being an experimentalist, but I suspect lots of people who do behavioral experiments are drawn to the theorizing and storytelling parts while merely tolerating the statistics.
Useful post, thank you.
In clinical research we have tools and checklists coming out of our ears. And they are, for the most part, very useful tools and checklists. But there’s nothing forcing anyone to use them properly. Papers get littered with key words to tick a box, while the content betrays the authors’ cynicism and/or lack of understanding of what those words mean (and the peer reviewers’ or journal’s failure to fix it).
When the so-called replication crisis exploded (2011?), I was initially bemused. All these psychologists publishing things about statistics that we’d known for decades as if they were shiny and new. But then I started to see psych papers from the 1970s referenced, showing that psychology also worked all this out decades ago. And then I remembered the pitched battles we have in clinical research to simply hold the methodological line, let alone advance it. We’re doing a little bit better only because the nature of clinical research makes the ethical case for sound methods a lot stronger.
I’d love to break the cycle of tools developed -> corner-cutting -> disaster/scandal -> more tools developed. I don’t know if it is possible in this world, given the financial and/or career incentives to do harm. But I refuse to succumb to nihilism, tempting as it so often is.
>>”most people who do it presumably want to”
Precisely the problem.
This is easier said than done, but I try to start with the spirit of the law before pointing to its letter. People are terrified of statistics. I teach medics, who had to have a strong maths background to get into med school, and they are (almost) all terrified of statistics. It helps a lot if they understand the why before they get to grips with the how.
When I have the time luxury of a class to deliver, I get them to work out the methods for themselves, with small group work and/or guided questioning. Then when you put up the slide showing that they got it right just by thinking it through, it gives them a lot of confidence. You can put all the detailed head-hurty stuff in handouts they can consult, if and when they need to apply the methods they now understand.
In the real world of research, the ideal is applied statisticians, with experience in the specific field of application, on every project, involved from the start. But that isn’t going to happen any time soon*. So I try to tailor advice in a similar way. Talk through the project and get them to think about what they’re trying to achieve, with a mix of guided questions and helpful examples. Then find them some practical guides on how to actually do it.
If what they’re trying to do (in context with why they’re trying to do it) cannot be simplified enough, or left to software to do for them if they get the dataset right, I’ll try to work out how they can get the ongoing support that they need. Again, easier said than done.
Thanks again.
*although all the major funders of clinical research do insist on it these days, and some of them require a named data manager too
This is a great example. I felt a little lost reading this entry but your example of golf and gold swings really helped me understand this. While everyone claims that there are these main things you need to do, but you can implement something else that works. As well as some completely foreign entity can mess it all up.
Just want to say that I feel similarly, and it’s nice to hear someone saying this.
Independent replication could be ritualized. Imagine the psychology/cancer reproducibility projects for every study. If its worth funding once, its worth funding twice.
But people apparently don’t like the one ritual that would actually work. Theyd rather use prediction markets or misinterpret p-values. I think it just comes down to a critical mass of researchers don’t actually like science.
This sounds so obnoxious, I know. But really, try to get independent replications going and see what happens. This is the reality anyone can observe.
But the resources we’d be wasting on studies that we weren’t likely to learn from in the first place. Some would say we’re already replicating too much without thinking about the value of the information we gain.
I mean, we could use sampling from among studies as a method to come up with ideas of research output quality without spending anything close to literally double… so what if we added 10% to the budget for NIH and spent the 10% on replication of random bio-medical studies from the last few years? I think you’d find far far too little information was actually gained in those original studies.
Spot on as usual. I always enjoy reading your thoughts, thank you.
The work of you and others that push for higher standards in science is really commendable, but I can’t help but feel extremely depressed that we need to babysit adults that are supposed to be the smartest guys around.
Thanks so much. Great perspective, as usual.
My day job is consumed with reviewing machine learning models for bias and equity issues. We deploy hundreds of models a year from a mix of on-shore, off-shore, and contracted talent as well as vendor built solutions.
We face the exact same issues trying to get modeling teams to follow the checklists and guidelines. We cannot implement “do these three things” because that appears like a moving goalpost to the teams doing the work. Our best would be ordering our checklist in terms of possible harms.
At least I can wield a big stick – our reviews are a compliance process that originates from our legal staff. Teams must pass our guidelines or they cannot deploy their models
Aren’t you presenting a bit of a false dichotomy? Essentially you argue that (1) either we start with large sample sizes etc. and move to model specification and heterogeneity or vice versa and (2) that the vice versa order is the better approach.
Shouldn’t both aspects be essential components to good science? Without sensible sample sizes, explicit RDFs or separation between exploratory/confirmatory analysis, appreciation of heterogeneity will not save you. While there are clear issues with model mispecification, weak theory and so on.
Relating this to your analogy, I don’t think there is any set of simple things that will make a learner into a great writer. My view, instead, is that any model in which the learner picks up a few things—whether to do with model specification, heterogeneity, measurement etc—and then stops the cycle will inevitably suffer from problems. So constant iteration is needed.
“Aren’t you presenting a bit of a false dichotomy?” “Shouldn’t both aspects be essential components to good science?”
Nowhere in the post am I arguing that one or the other will suffice.
“I don’t think there is any set of simple things that will make a learner into a great writer.” Me neither! That’s one of the points I’m making. How do you approach reform when you know iteration is essential, and even with iteration its still going to be really hard. At the very least, you should be careful not to imply that a few simple tricks alone are going to be enough to solve the problems.
“At the very least, you should be careful not to imply that a few simple tricks alone are going to be enough to solve the problems.”
Excellent! Pre-reg etc are not substitutes for sound science. If you:
(1) make inaccurate assumptions;
(2) use fuzzy, or even irreproducible measurement methods (e.g., using proxies that rely on even more assumptions)
(3) and top those two off with dodgy analytical methods;
no amount of pre-registration can save your sorry behind! :)
Happily, it’s realtively easy to conquer (3) – hook up with a sensible stats prof and spend 4-6 yrs on a PhD and you’ll get there. However, the first two just aren’t as easy as a PhD, since there is frequently no way to enumerate all the implicit assumptions in many of the questions social scientists are asking; and almost never a way to directly measure hte variables of interest.
I like your writing analogy Jessica. Perhaps you could take it a bit further and bring in the concept of resolution. Poor writing and poor science are analogous to an image with low resolution: often even the main themes are barely discernable. Improving both writing and science is like increasing the resolution of one’s camera: as the number of pixels increases, the image the camera renders becomes more clear – more accurate – and richer in details. I especially like the analogy to science – progress in science is driving an increase in the resolution of our understanding of hte universe. But it does not come easy or fast.
Of possible historical interest, spanning 1998-2010:
“The hard issue is whether the methods of large parts of social science are bogus, phony, pseudoscientific. They are. The other hard issue is whether there are better methods [adapted] to the important parts of social science. There are.” (171)
Clark Glymour, “Social Statistics and Genuine Inquiry: The Case of _The Bell Curve_,” in _The Mind’s Arrows: Bayes Nets and Graphical Causal Models in Psychology_ (The MIT Press, 2001) 171-204. [An earlier version was published in 1998 in the journal, _Philosophy of Science_ and contains some interesting reports on the sociology of Statistics and the social sciences that is not in the book.]
“The conjunction of statistical and probabilistic reasoning is too difficult for people, even people who claim to be experts. … Causal analysis in many social sciences is effectively computerized rituals with factor analysis and regression. Better computational procedures are available for the problems, but bad methods persist by tradition and because bad methods give definite answers where better founded procedures would report uncertainty. Uncertainty is hard to publish. But no computerized methods warrant failing to think about alternative explanations the computer could not have found or failing to check them out where one can.” (136-137)
Clark Glymour, “Where Software Goes to Die,” in _Galileo in Pittsburgh_ (Harvard University Press, 2010) 134-138.
For some additional information about Glymour’s work (with others), see: https://www.cmu.edu/dietrich/news/news-stories/2020/august/tetrad-sail.html.