The above image, taken from a site at the University of Virginia, illustrates a problem with political punditry: There’s a demand for predictions, and there’s no shortage of outlets promising a “crystal ball” or some other sort of certainty. Along these lines, Elliott Morris points us to this very reasonable post, “Poll-Based Election Forecasts Will […]
Archive of posts filed under the Decision Theory category.
Getting all negative about so-called average power
Blake McShane writes: The idea of retrospectively estimating the average power of a set of studies via meta-analysis has recently been gaining a ton of traction in psychology and medicine. This seems really bad for two reasons: 1. Proponents claim average power is a “replicability estimate” and that it estimates the rate of replicability “if […]
On deck through Jan 2021
This should keep you busy through the end of the year . . . Many of these posts were originally scheduled earlier but then got bumped because of coronavirus and other topical material. The typical set and its relevance to Bayesian computation Getting negative about the critical positivity ratio: when you talk about throwing out […]
Thinking about election forecast uncertainty
Some twitter action Elliott Morris, my collaborator (with Merlin Heidemanns) on the Economist election forecast, pointed me to some thoughtful criticisms of our model from Nate Silver. There’s some discussion on twitter, but in general I don’t find twitter to be a good place for careful discussion, so I’m continuing the conversation here. Nate writes: […]
Negativity (when applied with rigor) requires more care than positivity.
Tyler Cowen writes: Avoid criticizing other public intellectuals. In fact, avoid the negative as much as possible. However pressing a social or economic issue may be, there is almost always a positive and constructive way to reframe your potential contribution. This also will force you to keep on thinking harder, because it is easier to […]
Would we be better off if randomized clinical trials had never been born?
This came up in discussion the other day. In statistics and medicine, we’re generally told to rely when possible on the statistically significance (or lack of statistical significance) of results from randomized trials. But, as we know, statistical significance has all sorts of problems, most notably that it ignores questions of cost and benefit, and […]
Further debate over mindset interventions
Warne Following up on this post, “Study finds ‘Growth Mindset’ intervention taking less than an hour raises grades for ninth graders,” commenter D points us to this post by Russell Warne that’s critical of research on growth mindset. Here’s Warne: Do you believe that how hard you work to learn something is more important than […]
“To Change the World, Behavioral Intervention Research Will Need to Get Serious About Heterogeneity”
Beth Tipton, Chris Bryan, and David Yeager write: The increasing influence of behavioral science in policy has been a hallmark of the past decade, but so has a crisis of confidence in the replicability of behavioral science findings. In this essay, we describe a nascent paradigm shift in behavioral intervention research—a heterogeneity revolution—that we believe […]
Priors on effect size in A/B testing
I just saw this interesting applied-focused post by Kaiser Fung on non-significance in A/B testing. Kaiser was responding to a post by Ron Kohavi. I can’t find Kohavi’s note anywhere, but you can read Kaiser’s post to get the picture. Here I want to pick out a few sentences from Kaiser’s post: Kohavi correctly points […]
No, I don’t believe that claim based on regression discontinuity analysis that . . .
tl;dr. See point 4 below. Despite the p-less-than-0.05 statistical significance of the discontinuity in the above graph, no, I do not believe that losing a close election causes U.S. governors to die 5-10 years longer, as was claimed in this recently published article. Or, to put it another way: Despite the p-less-than-0.05 statistical significance of […]
The value of thinking about varying treatment effects: coronavirus example
Yesterday we discussed difficulties with the concept of average treatment effect. Part of designing a study is accounting for uncertainty in effect sizes. Unfortunately there is a tradition in clinical trials of making optimistic assumptions in order to claim high power. Here is an example that came up in March, 2020. A doctor was designing […]
Embracing Variation and Accepting Uncertainty (my talk this Wed/Tues at a symposium on communicating uncertainty)
I’ll be speaking (virtually) at this conference in Australia on Wed 1 July (actually Tues 30 June in our time zone here): Embracing Variation and Accepting Uncertainty It is said that your most important collaborator is yourself in 6 months. Perhaps the best way to improve our communication of data uncertainty to others is to […]
This one quick trick will allow you to become a star forecaster
Jonathan Falk points us to this wonderful post by Dario Perkins. It’s all worth a read, but, following Falk, I want to emphasize this beautiful piece of advice, which is #5 on their list of 10 items: How to get attention: If you want to get famous for making big non-consensus calls, without the danger […]
No, there is no “tension between getting it fast and getting it right”
When reading Retraction Watch, I came across this quote: “There is always a tension between getting it fast and getting it right,” said Dr. Marcia Angell, another former editor in chief of the New England Journal of Medicine. “I always favored getting it right. But in the current pandemic, that balance may have shifted too […]
Do we really believe the Democrats have an 88% chance of winning the presidential election?
OK, enough about coronavirus. Time to talk about the election. Dhruv Madeka starts things off with this email: Someone just forwarded me your election model (with Elliott Morris and Merlin Heidemanns) for the Economist. I noticed Biden was already at 84%. I wrote a few years ago about how the time to election factors a […]
(Some) forecasting for COVID-19 has failed: a discussion of Taleb and Ioannidis et al.
Nassim Taleb points us to this pair of papers: On single point forecasts for fat tailed variables, by Nassim Taleb Forecasting for COVID-19 has failed, by John Ioannidis, Sally Cripps, and Martin Tanner The two articles agree in their mistrust of media-certified experts. Here’s Taleb: Both forecasters and their critics are wrong: At the onset […]
Advice for a yoga studio that wants to reopen?
This post is by Phil Price, not Andrew. My 79-year-old mom likes to go to yoga classes, although of course she has not done so in months. Her favorite yoga place is cautiously reopening — they’ve had a few sessions with just eight or ten people in a rather large space (I’m going to guess […]
Why X’s think they’re the best
Commenter Alex pointed out this excellent post, Why Doctors Think They’re the Best, by Scott Alexander, who writes: Ninety percent of drivers think they’re above-average drivers, ninety percent of professors think they’re above-average professors etc. The relevant studies are paywalled, so I don’t know if I [Alexander] should trust them. . . . But I […]
The seventy two percent solution (to police violence)
And now it is your turn, We are tired of praying, and marching, and thinking, and learning — Gil Scott-Heron So. It turns out that Gil Scott-Heron was right and he was wrong. We once again, during a time of serious social inequality and political upheaval, sent whiteys to the moon (ish). On the other hand, the […]
Vaccine development as a decision problem
This post by Alex Tabarrok hits all the right notes: At current rates, the US economy is losing about $40 billion a week. Thus, if $20 billion could advance a vaccine by just one week that would be a good deal. . . . It might seem expensive to invest in capacity for a vaccine […]