Wed 2 Oct 9:30am at Bin Yu’s research group, 1011 Evans Hall, University of California, Berkeley:
Toward reproducible research: Some technical statistical challenges
The replication crisis in social science is not just about statistics; it has also involved the promotion of naive pseudo-scientific ideas and exaggerated “one weird trick” claims in fields ranging from embodied cognition to evolutionary psychology to nudging in economics. In trying to move toward more replicable research, several statistical issues arise; here, we discuss challenges related to design and measurement, modeling of variation, and generalization from available data to new scenarios. Technical challenges include modeling of deep interactions and taxonomic classifications and the incorporation of sampling weights into regression modeling. We use multilevel Bayesian inference, but it should be possible to implement these ideas using other statistical frameworks.
Thurs 3 Oct 10:30am at the Stanford classical liberalism seminar, room E103, Stanford Graduate School of Business:
The political content of unreplicable research
Discussion of the replication crisis in the social science has focused on the statistical errors that have led researchers and consumers of research to overconfidence in dubious claims, along with the social structures that incentivize bad work to be promoted, publicized, and left uncorrected. But what about the content of this unreplicable work? Consider embodied cognition, evolutionary psychology, nudging in economics, claimed efficacy of policy interventions, and the manipulable-voter model in political science. These models of the world, if true, would have important implications for politics, supporting certain views held on the left, right, and technocratic center of the political spectrum. Conversely, the lack of empirical support for these models has implications for social science, if people are not so arbitrarily swayed as the models suggest.
The two talks should have very little overlap—which is funny, given that I’ll probably be the only person to attend both of them!
In preparation for both talks, I recommend reading the first three sections of our piranha paper.
The Stanford talk is nontechnical, talking about the social science and policy implications of the replication crisis, and I want to convey that the replication crisis isn’t just about silly Ted talks; it also has implications for how we should understand the world.
The Berkeley talk is for statisticians, talking about how the roots of and solutions to the replication crisis are not just procedural (pregregistration etc.) or data-analytical (p-values etc.) but also involve measurement, design, and modeling.
Interesting the libertarians at Stanford would invite you to speak on the relevance of your and others’ work for social theory! This surprises me, especially in light of the debunking of quasi-libertarian anti-Covid-public-health “science” at Stanford itself. Two thoughts leap to mind. (1) As libertarians, they are skeptical of research-ish claims for the efficacy of government intervention, which is good of course so long as it doesn’t pair with credulousness regarding the performance of market economies in the absence of intervention — a sort of one-way street fallacy, no? (2) They may be under the influence of Hayek, whose political views were partly formed by his revulsion at the positivist enthusiasm of 1920s Vienna. He doubted the ability of any rationalist project to comprehend, much less improve upon, the genius of self-organizing systems. That’s useful skepticism too, but he and his acolytes carried it too far, awarding self-organization a halo it doesn’t deserve.
Peter:
I think of this in the same way I’d think about a statistics seminar which might invite applied people, theory people, Bayesians, anti-Bayesians, etc. The general topic is statistics, and speakers offer many different perspectives. This Stanford seminar is about “interactions among individuals, corporations, markets, government, and civic institutions in a free society,” which is pretty broad, so I guess it can include a quantitative political scientist such as myself along with people whose activities I absolutely can’t stand, such as Cass Sunstein and Alan Dershowitz, who I can only assume regaled the audience with you-had-to-be-there stories of Henry Kissinger and Jeffrey Epstein.
What part of the public health response do you consider scientific?
Ie, something like:
1) group A observes something
2) group B verifies
3) explanations for this consistent observation are guessed
4) predictions are deduced from the various explanations
5) predictions are then compared to new data
6) the models with the best predictions are kept and refined
Do a search on this website for “Great Barrington” or “Ioannidis” (post-2020). There are lots of posts about egregious errors in public-health related studies by Stanford faculty who opposed various responses to the pandemic.
How about the actual public health responses that affected people though?
I don’t want to get into a general debate over Covid policy. I supported some of it and thought some was really misguided/harmful. The issue here is faulty data work by a few researchers with a political ax to grind. Nonreplicable findings are the topic of Andrew’s talk.
No need for agreement or debate. Id just like to know which aspect you consider to be “science” so I can look closer. I won’t respond further.
I followed it pretty closely and feel confident zero science was performed.
Lots of preliminary wild speculation and NHST bizarro-science though.*
This was as expected since, like in the social sciences, that is standard medical research behavior.
* Sometimes there was a piece of cheese, others some bread, maybe a grill was used, but never did it all come together to make a grilled cheese sandwhich.
How about the actual public health responses that affected people though?
In addition to the aforementioned Ionnidis, and specific to Stanford, Google Batacharya, Levitt, or Atlas and you’ll find plenty o’ evidence of non-replicable science that not only “affected people,” but also directly influenced public health policy.
If you’d like to branch out beyond Stanford you could try Makary or Ladapo. Since you’re very motivated, I’m sure you’ll come up with much that’s of interest.
Sorry, two “t’s” in Bhattacharya
Here’s what this guy is doing and why it is totally useless to answer him. “Toxic tactics” from Adam Kucharski’s Substack
https://kucharski.substack.com/p/toxic-tactics
It seems there is no part of the public health response that comes to mind as scientific.
Critiques of the response come to mind as definitely unscientific, but in the form of names and political affiliation rather than topic/evidence/reasoning. Fascinating.
Further, such questions are toxic. Thus they can be safely ignored without further thought.
I’d guess there is no definition of “science” that includes the implemented public health policies but excludes the critiques.
Anoneuoid –
It seems there is no part of the public health response that comes to mind as scientific.
If you draw the line between “scientific” and “unscientific” in a binary and simplistic appeal to perfection, then of course no one can meet your impossible standard. Which then raises the interesting issue of why you even bothered to ask the question, since you were bound and determined to just confirm your bias no matter how anyone responded. What is the appeal to tautological reasoning? It’s very common, so it must serve some kind of a purpose.
When I did medical research they told me it was impossible for my topic (roughly, brain injuries and learning) too. Then I found a bunch of stuff from pre-1940 already doing it.
So I don’t buy that.
It is the low standards themselves that make it effectively impossible, by sucking up all the funding/mindshare in a gresham’s law-type effect.
Anoneuoid –
Because people get annoyed by this…one more response.
Here’s what you said above:
I followed it pretty closely and feel confident zero science was performed.
I think the basic question of what public health policy merits a label of scientifically rigorous is interesting and important – no doubt. That question gets to the root of much of what takes place at this blog. But the answer is always going to be subjective to some degree and reducing it to a simplistic binary as you do – particularly in the context of COVID public policy where you’re highly invested in a particular framing – is just a tautological exercise. Your conclusion is already formulated. So advance your arguments – they’re sometimes interesting. But don’t bother framing it as a rhetorical question for which you’re pretending to seek an answer.
I didn’t make the distinction, Peter Dorman did.
Anyway, another complete waste of time trying to learn from people who thought the covid response was valid, in any way.
Seems there is no science behind those beliefs at all, just tribalism.
Argh. Despite being local, these times/dates don’t work, and I’m going to miss the talks :(
(Otherwise I probably would have been the other person to attend both.)
Andrew,
Do you know if your talks will be available on YouTube?
Regards,
Pablo