Art Owen writes:
I saw the essay, “Nothing Scales,” by Jason Kerwin, which might be a good topic for one of your blog posts. Maybe a bunch of other people sent it to you already.
He seems to think we just need more and better data and methods to get things to generalize/scale. It’s not clear to me that we’ll get enormously better data per subject on education or behavior. Maybe we will get better sets of subjects (more coverage) in a more complex and expensive study.
The post in question is by an economist who is emphasizing the importance of varying treatment effects. This is something that people been talking about for awhile, but, as with many things in statistics, this is something that we each have to rediscover on our own. It’s said that the best way to learn something is to teach it, so it’s good to see Kerwin’s discussion, which he develops in the context of a real example. And I appreciate that he refers to 16.
Just a couple minor things. Kerwin writes:
Treatment effect heterogeneity also helps explain why the development literature is littered with failed attempts to scale interventions up or run them in different contexts. Growth mindset did nothing when scaled up in Argentina. Running the “Jamaican Model” of home visits to promote child development at large scale yields far smaller effects than the original study. The list goes on and on; to a first approximation, nothing we try in development scales.
Why not? Scaling up a program requires running it on new people who may have different treatment effects. And the finding, again and again, is that this is really hard to do well. . . .
I’m with him on the importance of varying treatment effects, but, when it comes to explaining why estimated effects don’t replicate at their published magnitudes, I think he’s missing the big point that published estimates tend to be overestimates because of the winner’s curse (selection bias); see for example here. Also he writes, “None of the techniques we use to look at treatment effect variation currently work for non-experimental causal inference techniques.” That’s not true at all! Plain old regression with interactions works just fine, or you can break out the nonparametrics as with Hill (2011).
Again, I like Kerwin’s main point, which is that when considering how a treatment will scale in the real world, it’s important to think about treatment effect variation, not just as a mathematical concept (correcting for “heteroscedasticity” or whatever) but substantively. I also agree with what Caroline Fiennes writes in comments, that it’s important to know what is the cost of an intervention and what exactly the intervention is.
The reason treatments almost never scale is that one flippin’ study isn’t enough to find out if they actually work or not!!! COME ON!!! WAKE UP!!!!
We’re not testing the ideal freakin’ gas law in a closed system! Human behavior has dozens and dozens of dimensions of variation. Sorry, you’re never going to cover that in one or two or three or four studies. It took *1500 YEARS* of scientific inquiry to establish the nature of a simple thing like the solar system, but sociologists seem to think they’re going to remake history overnight with NHST and a spreadsheet. PUHLEEEEZ!!!!!!
This is what kills me about this topic, this blog, these discussions. You guys are dinking around with magical incantations. Most of the time you’re worrying about whether to say “abracadabra” or “alakazam”. None of which would matter if you were a privately funded magic NGO playing games in your basements, but no, you’re wasting millions of dollars of public money blowing wind and actually harming people with obviously wrong “interventions”, the effects of which are nothing other than figments of your imaginations.
Question – Let’s say you do the intervention on your sample and you run your regression model with varying treatment effects for individuals (i.e. varying slopes and intercepts). In fact, you could include varying effects for many things in the model, not just the treatment. Something like this in rstanarm or brms:
outcome ~ 1 + treatment + x2 + x3 + (1 + treatment + x2 + x3 | id)
From the model output you can obtain posterior samples for the standard deviation of these varying slopes and intercepts, so you can easily get uncertainty intervals on this variation.
So, if you wanted to get an idea of what happens when you scale up the same intervention, couldn’t you simulate fake data from the above model or at least using the parameter estimates, and see how it might vary from the smaller intervention? For example, in a worst case scenario, maybe the variability in the treatment effect (standard deviation of the varying slope) is on the upper end of the 95% UI. Then couldn’t you simulate data using that upper estimate to see what might happen in a large fake dataset where treatment varied a lot?
Of course, this would be ignoring any kind of other problems (like watered down intervention) which were discussed in the article.
I didn’t understand this proposal. You’d need a lot of repeated measurements from each person (id here) to fit a model like that such that the variance components are reasonably well estimated. I don’t think that that’s usually possible to do in the kinds of scenarios discussed here. In psych experiments, sure.
The large number of dimensions of variation in human behavior immediately makes it clear why sociologists continually fail to make the human condition better – and frequently make it worse – while markets continually succeed to make the human condition better. Sociology is top-down, markets are bottom up. Sociologists emphasize what they think is good for people. Markets make no judgement but instead allow infinite “experiments”. The human(?) impulse to imitate amplifies the successful variations until they become endemic at their level of effectiveness.
How long would it take to reach modernity if we provided a primitive society – say, 30,000 years ago – with a sociologist? :) The answer is “never”. The sociologist is the problem that has to be overcome, not the solution. The sociologist is just the modern Western version of a shaman.
Jim:
I don’t disagree with the thread of your comment . . . but I don’t think “sociologist” is the right term for you to use here. Sociologists often study bottom-up aspects of human interactions. The people who are doing these studies to try to make things better from the top down are policy analysts who are typically economists.
In defense of top-down government interventions, let me just make the usual argument that the government is going to be doing something, so it makes sense to try to do something more rather than less useful. There’s a separate argument about appropriate size of government, tax rates, redistribution, etc., but conditional on some level of government spending, it makes sense to do it wisely. And, even beyond spending, there are other policy issues such as rule-making.
I think jim’s line of reasoning is different – particularly the first comment he posted. He clearly doesn’t believe academics are worth listening to – they over-reach, distort, and contemplate their navels while the real world (at least where markets are working) get on with achieving things. Yes, my words, not his. I have some sympathy with that line of reasoning. Academics (whether sociologists, economists, or statisticians) are not always well-meaning, and often get things wrong due to incompetence even if they are trying to do good work. But, it seems to me that jim has been following too many talk shows and listening to too much of the populist propaganda. If we can never learn anything from one study, let alone a small handful, then of course we shouldn’t bother studying anything. We should just make the decisions that feel good – and the magic of markets will ensure that the good ideas survive and the poor ones fail.
I won’t bother to debate this. I’m not even sure how much I believe it is wrong. But I come to this blog, and spend most of my work time and thinking because I believe seeking truth matters and is a worthwhile pursuit. Markets are mindless. They do a great job of accomplishing the satisfaction of certain human desires – better than all alternatives that I know of. But markets don’t value truth unless people do.
Dale:
I guess that most academics aren’t worth listening to, just as most people aren’t worth listening to! But academia has some advantages, partly the freedom to see different connections, partly the benefits of specialization, partly selection of who wants to do academic research and can jump through the required hoops to get the job. I just thought it was funny that Jim used the word “sociologist” in his comment. Sociologists tend to have pretty extreme left-wing views, but I think they’re largely on his side when it comes to having a bottom-up perspective (even if they disagree with him regarding optimal tax rates, the appropriate role of the government, etc.).
Thanks to both of you for your thoughtful comments. I appreciate your patience with my occasional exasperation.
Andrew: I’m sure I’m not worth listening to sometimes or even often! I agree that many of the “appliers” are policy people. But not all. Some social scientists are science-oriented (like yourself), others are policy oriented and seem to be in the social sciences exclusively for that purpose (hero meme).
Dale: I don’t contend you can’t learn anything from one study. I contend that you can’t take a single poorly controlled (by definition) study of a complex phenomenon simply apply its conclusions to the world and expect them to hold up. This is not a controversial position. Medical interventions are overseen by several federal agencies, approved only after many studies, and frequently rejected for lack of efficacy after many studies suggesting they would work.
Also, regarding markets: I agree that rules are necessary. Rules that enforce basic standards of human conduct and allow the enforcement of business contracts are fundamental to successful markets. Private property rules, for example, aren’t a requirement for human society. It just turns out that for human wealth and prosperity they work better than other sets of rules.
Thanks again to both of you for your patience.
Jim,
As an older person who worked in environmental science, I have a less favorable view of markets. When I was young, shortly after WWII finally ended the Great Depression, I heard a lot about the irrationality of the market: oranges being burned to increase prices while lots of people were gong hungry, and that kind of thing. Then, in the 1960s, people started paying attention to the environmental harm from market-driven economic activity. A lot of that got dealt with by regulation, but a lot didn’t. Now, ~60 years after good data started coming in on increasing atmospheric carbon dioxide, we are starting to take climate change seriously, but a lot of damage has already been done or is baked in. What keeps me awake at night is the thought that climate change will drive a lot of migration, some of which is already happening, and this in turn will lead to right-wing populist responses where migrants go, with bad consequences for democracy. Markets seem like a great way to fine-tune economic activity, but the externalities are too great to trust them very far.
John, you seem to conflate free markets and environmental damage. It seems to me that any economic system that encourages development and growth would have similarly destructive effects. Read about Soviet whaling practices sometime, populations were hunted to near extinction without necessity merely to match magical numbers included in a 5-year plan.
David,
I agree that bad planning can make economic growth as destructive as markets do, and I don’t have a good answer for the problem. I would like to think that democratically guided planning could do better, but people seem to like stuff too much.
Pure markets come with incentives to privatise profits and socialize costs, and one way to do the latter is to not care about the environment.
Markets are primarily set up to benefit individuals.
I come not to praise sociologists but to try to bring some sanity to this ludicrously over-praising of “markets.” Markets are great, don’t get me wrong: invisible hand, optimization, incentives, yada yada, that’s all great.
But “markets” also gave us tens of millions of cigarette addicts, followed by tens of millions of opioid addicts. Hey, sell the people what they want to buy, what could be wrong with that, amirite?
“Markets” gave us a Cuyahoga River that caught fire seven times in the nineteen-sixties, to name just one of a many . Over the years they have given us financial crashes and crises, quack medicines (including some that were dangerous), Ponzi schemes, and of course unsafe working conditions, child labor, etc.
I would not try to defend every regulatory response to market failures and abuses, but I do want to mock the suggestion that unfettered free markets are the solution to everything, if in fact that suggestion is being made or implied.
Phil:
Not to get all Milton Friedman on ya, but I’ve read that massive cigarette addiction in the U.S. came in part from exposure in the military. Also, lots of smoking in China. I’d attribute this to various things, including the appeal of smokes and also greed, but not “markets,” exactly.
I agree with your general point that people will pay for, and other people will market, all sorts of things with bad long-term consequences.
Funny, I almost didn’t mention cigarettes because of the complication of free cigarettes being given out by the government in WWII. All those nee addicts. But WWII only lasted 4 years (for the US).
But ok, I could have chosen another example, it’s not like they’re hard to come by.
Jim said,
“markets continually succeed to make the human condition better”
What evidence do you have to support this statement? (Be sure to include your criteria for “better”!)
Martha –
> What evidence do you have to support this statement?
+1
I’d like to see the answer to that…
I’m particularly curious to see the equation that distinguishes the mechanism of “markets = modernity” compared to the mechanisms of “free education = modernity” or “antibiotics = modernity” or “women getting the vote = modernity,” etc.
> How long would it take to reach modernity if we provided a primitive society – say, 30,000 years ago – with a sociologist? :) The answer is “never”.
I’m pretty sure your tribe/society without a strong, central, authoritarian leadership would have been conquered quickly in pretty much every era of history or pre-history before the advent of the printing press.
We reached “modernity” because money first took power from the moral authority (the church) and then took power from those who felt responsible for a society because their children would inherit it (the nobility), and look what that’s done to war and hunger in the world!
Most of the important interaction variables aren’t available at all… only observables (age, race, gender, education) correlated with them. And even where they are available (or highly enough correlated with the standard observables) studies are often far too underpowered to adjust for all the interactions. Hierarchical Bayesian methods help reduce the curse of dimensionality, but you know what doesn’t scale up? People smart enough to run these studies.
So you’re left with underpowered studies, which, combined with the significance filter, produce the crap Andrew’s been talking about since I started reading this blog, and whose end is nowhere in sight.
Happy Thanksgiving!
Jonathan:
Too bad you weren’t reading the blog back in the early years, when I was mostly ranting about bad graphics and ax-grinding journalists who couldn’t or wouldn’t distinguish between individual and aggregate patterns. Those were the days.
I suppose plain old regression to the mean is one way of putting the selection effects. You have a teacher do X, and it works because he got lucky or is an unusually talented teacher or was having a good year, and you publish. You have 1,000 teachers do X, and nothing happens.
Economics provides a few special reasons why scaling fails.
1. If you are the first to do X, you profit, but then when everybody else does it, you all drive prices down and profits are back to 0 (tho consumers win).
2. If you try using input X, it works and you profit, but when you scale up 1,000 times, you drive up the price of X, and your profits go back to zero– only the owners of X benefit.
3. If you do X on a small scale, it works because you can manage it, but when you scale up 1000 times, you can’t manage everything by yourself and it all falls apart.
4. If you apply improvement X, it works because you start with the situation where it works best, but when you repeat it 100 times you run into diminshing returns as you apply it to less and less applicable situations. (The Wal-Mart problem: their first stores were in great locations, but they ran out of great locations, so their average profit per store (not total profit) fell.
Thanks for taking a look at my post, and for sharing it. Two thoughts on Andrew’s comments:
1. > I think he’s missing the big point that published estimates tend to be overestimates because of the winner’s curse (selection bias); see for example here.
I agree that this is a problem in general, I just don’t think it’s the whole problem. At least in some cases, if you do exact (or close to exact) replications of these programs in essentially the same context, you get close-to-identical results. One example of this is the No Lean Season program; the original paper that motivated the program contained a replication where they ran the RCT a second time and get extremely similar results. Another is the study I referenced on scaling up a literacy program in Uganda. We get almost the exact same results in the second year of the study, on a brand-new set of schools that were randomized to the same treatment.
This is the point I was trying to make in this part of my blog post:
“Note that this is a very different challenge from the “replication crisis” that has most famously plagued social psychology. The average treatment effect of appointments in our study matches the one in the other study I mentioned above, and the original study that motivated No Lean Season literally contains a second RCT that, in part, replicates the main result.”
A minor point on this: I think the link in “see for example here” is broken; I cannot download the file.
2. > Also he writes, “None of the techniques we use to look at treatment effect variation currently work for non-experimental causal inference techniques.” That’s not true at all! Plain old regression with interactions works just fine, or you can break out the nonparametrics as with Hill (2011).
This is a fair point, and that sentence in the original blog post is overstated. A modified statement that is correct is that as far as I know, neither the classical Fréchet-Höffding (FH) bounds on the variance of the treatment effects, nor the recent work by economists on using ML to examine how effects effects vary systematically covariates, have been extended beyond randomized experiments. I imagine that extending the ML methods to non-randomized settings is pretty straightforward. I am less clear on what to do about the FH bounds.
Again, I really appreciate your reading and sharing my post and your comments.
This reminds me of Tal Yarkoni’s Generalizability Crisis paper: https://psyarxiv.com/jqw35/
In my field (psycholinguistics), nothing scales. Most psycholinguists build super elaborate theories of “human language processing”, looking almost exclusively at English. Then the moment you step away from this wonderful language and look at some obscure language like, say, German, these lovely theories come crashing down. We’re mostly unconcerned about this though, because we know that English is where it’s at. Whenever I submit a paper to a psych or psycholinguistics journal investigating a language like Armenian, Persian, German, Hindi, Chinese, Russian, or some such obscure language, a reviewer or editor (or both) will inevitably ask me to justify why I investigated this language. In one paper on Armenian, one reviewer praised us for investigating Albanian; that’s how irrelevant anything that’s not English is, it doesn’t even matter what you call it.
Even at conferences, people have stood up and asked me why such-and-such non-English language was the target of investigation. If English is under study, however, nobody ever questions that. Probably a lot of the theoretical developments in psycholinguistics are the result of overfitting to English data. I suspect this is because US scientists largely control the discourse through editorial positions and editorial boards in the top journals.