Antonia Krefeld-Schwalb, Eli Rosen Sugerman, and Eric Johnson write:
Policymakers increasingly rely on behavioral science in response to global challenges, such as climate change or global health crises. But applications of behavioral science face an important problem: Interventions often exert substantially different effects across contexts and individuals. We examine this heterogeneity for different paradigms that underlie many behavioral interventions. We study the paradigms in a series of five preregistered studies across one in-person and 10 online panels, with over 11,000 respondents in total. We find substantial heterogeneity across settings and paradigms, apply techniques for modeling the heterogeneity, and introduce a framework that measures typically omitted moderators.
I like this. It reminds me of our piranha paper but directly informed by empirical data.
The focus on treatment interactions—equivalently, variation of effect size—makes sense to me. It’s something I’ve been thinking about for a long time, for example this from 2005, this from 2014, this from 2015, this from 2023, and our recent paper on causal quartets for visualizing varying treatment effects. Also this paper from 2004 that I’m still chewing on.
But I haven’t thought so much about modeling (rather than just describing) variation in effects. so this paper by Krefeld et al. seems like an important step forward.
Magnitude and direction
Also I was struck by this statement from the paper:
Moderators are associated with effect sizes through two paths—effecting manipulation intensity and interacting with the effect of the manipulation.
This reminds me of something regarding education research—really, policy research in general—that I’ve been saying a lot recently but haven’t written down. The idea is that education is like a vector with a magnitude and a direction. The magnitude is how hard students work on their own or in small groups—those are the two scenarios where most of the learning gets done—and the direction is what they learn.
As teachers, we have two jobs. Job #1 is to motivate students to learn, that is, to increase the magnitude. Job #2 is to teach correct and useful things, that is, to get the direction right. My books are a mix of #1 and #2. To help with the magnitude, we try to structure the material to be clear, to smooth the path to learning and to give students lots of handholds: stories, examples, math, code, explanations, homeworks, all sorts of things. To help with the direction, we work hard to include useful material and to remove or to argue against ideas we think are counterproductive. For example, when we were writing BDA back in the early 1990s, a big idea in Bayesian inference was decision theory of Bayes estimators, and another idea was Bayesian null hypothesis testing. We put in very little on those topics in the book, and most of what we did put in was to explain why we weren’t putting it in. From the other direction, we pivoted the book around three chapters on hierarchical modeling, model checking, and the relation between design and analysis: to us, these were important concepts that students might otherwise not see.
As you can see just from the above paragraph, it’s a lot easier for me to think about direction than magnitude, which makes sense because I think I’m a much better statistician than a teacher, and indeed my teaching is best done not one-on-one but rather in this broadcasty way by exploring ideas through writing.
To get back to education research: I think most education interventions, at least the ideas that get tested in controlled trials, are focused on improving magnitude. The idea is that the subject-matter experts are supposed to get the direction right, and the education researchers work on the magnitude.
But this has implications for education research! What I’m calling “the magnitude,” which is motivation for students to work hard, figure things out, and learn on their own or with peers, is in large part an interaction between the teacher and student. That’s right, interactions again!
And not just education research. So many social interventions are ultimately about motivation.
This idea—that the most important part of a treatment is in its interaction with the people being treated—is in direct conflict with the dominant approach of thinking about causal inference, what I call the black-box or push-a-button, take-a-pill model of science. Something’s gotta give, and maybe this new paper by Krefeld et al. will take us a little bit in the right direction.
P.S. I could do without the trolley example—I’d be happy to never again hear about that fat guy (described in this article as “a large man wearing a backpack,” which I guess is the politically correct way to say “fat guy” now), but, hey, it’s their paper, they can use whatever examples they want!
This is really talking about model specification. Remember, even the simple t-test is really a linear regression.
We can reframe this as the model should have predictive skill, while “effect size” is just one parameter of the model.
In fact, we are free to choose any value we want for the “effect size”. Conventionally its chosen using OLS, but this is not strictly required.
If the model has no skill, who cares what value was chosen for that parameter? Eg: https://statmodeling.stat.columbia.edu/2025/02/28/the-r-squared-on-this-is-kinda-low-no/
Ie, predictive skill is paramount, *not* “effect size”. And certainly not “statistical significance” of effect size. Yet, stuff like drugs are approved using the later. Which is why the blockbusters all have NNT ~ 100 these days.
Out of topic: I’m slightly annoyed by the recent changes from statistics to data science, biology to life science, and now psychology to behavioural science. Is it just marketing following the very fashionable computer science? Or is it part of a general trend towards simplified language?
Andrew,
Interesting take. I like the separation of student motivation and content provision. I’m not on board with describing them as “magnitude” and “direction”. It doesn’t strike me as a great analogy but it’s useful to have labels I guess.
But I’m stumped by your contention that most educational interventions are an effort to influence magnitude / motivation. From what I’ve seen and read, to the extent that this is true at all, it’s true only at some rudimentary level, vaguely akin to participation trophies.
Most education interventions do one or all of three things beyond what “normal” teaching tries to do: 1) wrap the educational medicine in candy (videos, cartoons yada yada) in hopes that making it fun or entertaining will increase learning or attention; 2) deploy various techniques that amount to providing more assistance (“flipped” classroom); or 3) lowering the amount of learning required or increasing the reward for a given amount of learning (grade inflation). Normally all of this is measured through some kind of testing. I support testing but test results haven’t demonstrated much in the way of advances in learning for any method. Virtually everything reverts to the mean once it’s widely deployed.
Maybe I’m mistaken, but I can’t think of a single intervention I’ve heard of where the measurement was the amount of self-directed study time or where the intended effect was to increase the study time. Probably I’m wrong – surely there has at one time or another been some effort to measure this. But the “interventions” that make their way into actual application aren’t based on anything like that. Far and away the largest effort has been the relentless flogging of pre-K. Along with that has come all sorts of mumbo jumbo from CRT to changing school start times. None of this has shown any significant effect on learning and how those interventions constitute an effort to motivate students I just don’t get.
Maybe I’m mistaken about what the results of “motivation” should be? In my world, a more motivated student puts more time into their work. They are more determined to succeed. They measure their own progress and address shortcomings. What’s the alternative?
It’s interesting that I can disagree with so much of your comment but still agree to a large degree with what may be your main point – the outcome criterion to measire an educational system should be the amount and skill of a student’s self-directed learning. See my comment below.
Anon:
Your first two examples–making the lesson more fun to increase learning or attention; providing more assistance–fit into my category of increasing student motivation, the “magnitude” of my vector.
With all due respect (and while I couldn’t argue whether you’re a better statistician than teacher—I’ll take your word for it, though I suspect you’re excellent at both), I think your two-job framework misses something critical.
As teachers, we have two jobs. Job #1 is to motivate students to learn, that is, to increase the magnitude. Job #2 is to teach correct and useful things, that is, to get the direction right.
A teacher’s primary job, in my view, isn’t just to motivate students to learn (magnitude) and/or teach correct, useful things (direction). There’s a third job: helping students become better learners. This is closely tied to your two, but distinct—it’s about fostering meta-cognitive skills so they can adapt and grow independently. For most students, I think, this third job outranks the others in importance, though I agree there’s a hierarchy depending on the individual student.
That said, learning to be a better learner can be context-dependent. The subject domain might act as a moderator here, shaping how effectively a teacher’s efforts translate into meta-cognitive gains. It could also be a mediator in the sense that understanding of statistics can greatly enhance a student’s meta-cognitive skills.
Whoa! I almost – almost – agreed! I hestiate to give an endorsement beacause experience suggests that you probably actually mean something wildy different than to what I think you mean.
I have a problem with the idea that “the amount and skill of a student’s self-directed learning” should be the criterion for measurement. I suggested measuring this as a test to understand how “interventions” – which Andrew claims are largely intended to inspire motivcation – are working, not as an end in itself. The ultimate end is knowledge. Different students will have different results because some students have more experience and some students have more natural ability and some students have more of both. The net of these are tallied in the end by measuring knowledge. Ultimately what mixture yields the greatest knowledge is irrelevant. The point is to put the best-positioned students at the top of the opportunity list.
Joshua:
You write, “A teacher’s primary job, in my view, isn’t just to motivate students to learn (magnitude) and/or teach correct, useful things (direction). There’s a third job: helping students become better learners.”
I see what you mean, that we’re not just teaching skills and knowledge, we’re also teaching meta-learning skills. In my categorization, I was considering such meta-learning skills as part of the “correct, useful things” that we teach.
Consider this analogy. If you’re teaching a course in American politics, you’ll probably want students to also learn some statistics (to help them make sense of quantitative claims about politics) and to learn some programming (to help them implement statistics). Neither statistics nor programming is part of the subject matter of American politics but you still want students to learn them. Meta-learning skills (along with other general-purpose skills such as literacy) are like that. They’re part of what we teach; in my formulation, they’re part of “the direction.” And they can be taught in more or less useful directions too!
I think Joshua’s comment is more than what you portray as “part of the correct, useful things that we teach.” Increasingly, I think that teaching students to become better learners is almost the only thing we should be teaching (of course, this is overstating the case). But the world is full of answers – not all correct, nor are all equally useful. If a person can ask a good question, there are plenty of answers available. But they need to be able to ask good questions and they need to be able to evaluate various answers. All my experience with LLMs has reinforced that point for me. Of course, this doesn’t preclude teaching “correct, useful things” such as the statistics and programming you mention – and to some extent they are necessary if students are to learn how to ask good questions. But our current educational system is so lopsided, that I am inclined to overstate the case as an attempt to rebalance things.
While we all (teachers that is) strive to develop critical thinking skills, the vast majority of what we evaluate students on is their ability to provide answers (and in the worst cases, from memory on a time-limited exam). Incentives matter, and if that is what really “counts” then that is what students will concentrate on. They learn some bad habits along the way – such as inquiry ends when their brain has been emptied of what they know. Seeking ever more ideas is not rewarded as much as ideas they already possess. It is almost like we are training them to be replaced by AI in the future.
Sure, I’ve overstated the case, but as I said, I am trying to restore some balance. I feel like your response to Joshua’s point treats teaching students to become better learners as ancillary to the main jobs of motivation and correct useful things.
Dale:
I wasn’t saying that teaching students to become better learners is “ancillary to the main jobs of motivation and correct useful things.” I was saying that these skills are “part of what we teach”; that is, they are some of the “correct useful things” we want students to learn.
I’m having a bit of a problem with the link to the piranha paper …
Link fixed; thanks.
Ouf! (as they say in French) – light reading it ain’t; this’ll keep me busy during the weekend. Kudos for not dispensing any gratuitous swipes (we all know the usual suspects ;-)