I just saw this interesting applied-focused post by Kaiser Fung on non-significance in A/B testing. Kaiser was responding to a post by Ron Kohavi. I can’t find Kohavi’s note anywhere, but you can read Kaiser’s post to get the picture. Here I want to pick out a few sentences from Kaiser’s post: Kohavi correctly points […]
Archive of posts filed under the Decision Theory category.
No, I don’t believe that claim based on regression discontinuity analysis that . . .
tl;dr. See point 4 below. Despite the p-less-than-0.05 statistical significance of the discontinuity in the above graph, no, I do not believe that losing a close election causes U.S. governors to die 5-10 years longer, as was claimed in this recently published article. Or, to put it another way: Despite the p-less-than-0.05 statistical significance of […]
The value of thinking about varying treatment effects: coronavirus example
Yesterday we discussed difficulties with the concept of average treatment effect. Part of designing a study is accounting for uncertainty in effect sizes. Unfortunately there is a tradition in clinical trials of making optimistic assumptions in order to claim high power. Here is an example that came up in March, 2020. A doctor was designing […]
Embracing Variation and Accepting Uncertainty (my talk this Wed/Tues at a symposium on communicating uncertainty)
I’ll be speaking (virtually) at this conference in Australia on Wed 1 July (actually Tues 30 June in our time zone here): Embracing Variation and Accepting Uncertainty It is said that your most important collaborator is yourself in 6 months. Perhaps the best way to improve our communication of data uncertainty to others is to […]
This one quick trick will allow you to become a star forecaster
Jonathan Falk points us to this wonderful post by Dario Perkins. It’s all worth a read, but, following Falk, I want to emphasize this beautiful piece of advice, which is #5 on their list of 10 items: How to get attention: If you want to get famous for making big non-consensus calls, without the danger […]
No, there is no “tension between getting it fast and getting it right”
When reading Retraction Watch, I came across this quote: “There is always a tension between getting it fast and getting it right,” said Dr. Marcia Angell, another former editor in chief of the New England Journal of Medicine. “I always favored getting it right. But in the current pandemic, that balance may have shifted too […]
Do we really believe the Democrats have an 88% chance of winning the presidential election?
OK, enough about coronavirus. Time to talk about the election. Dhruv Madeka starts things off with this email: Someone just forwarded me your election model (with Elliott Morris and Merlin Heidemanns) for the Economist. I noticed Biden was already at 84%. I wrote a few years ago about how the time to election factors a […]
(Some) forecasting for COVID-19 has failed: a discussion of Taleb and Ioannidis et al.
Nassim Taleb points us to this pair of papers: On single point forecasts for fat tailed variables, by Nassim Taleb Forecasting for COVID-19 has failed, by John Ioannidis, Sally Cripps, and Martin Tanner The two articles agree in their mistrust of media-certified experts. Here’s Taleb: Both forecasters and their critics are wrong: At the onset […]
Advice for a yoga studio that wants to reopen?
This post is by Phil Price, not Andrew. My 79-year-old mom likes to go to yoga classes, although of course she has not done so in months. Her favorite yoga place is cautiously reopening — they’ve had a few sessions with just eight or ten people in a rather large space (I’m going to guess […]
Why X’s think they’re the best
Commenter Alex pointed out this excellent post, Why Doctors Think They’re the Best, by Scott Alexander, who writes: Ninety percent of drivers think they’re above-average drivers, ninety percent of professors think they’re above-average professors etc. The relevant studies are paywalled, so I don’t know if I [Alexander] should trust them. . . . But I […]
The seventy two percent solution (to police violence)
And now it is your turn, We are tired of praying, and marching, and thinking, and learning — Gil Scott-Heron So. It turns out that Gil Scott-Heron was right and he was wrong. We once again, during a time of serious social inequality and political upheaval, sent whiteys to the moon (ish). On the other hand, the […]
Vaccine development as a decision problem
This post by Alex Tabarrok hits all the right notes: At current rates, the US economy is losing about $40 billion a week. Thus, if $20 billion could advance a vaccine by just one week that would be a good deal. . . . It might seem expensive to invest in capacity for a vaccine […]
Laplace’s Theories of Cognitive Illusions, Heuristics and Biases
A few years ago, Josh “Don’t call him ‘hot hand’” Miller read Laplace’s classic book on probability theory and noticed that it anticipated much of the “heuristics and biases” literature (also called “cognitive illusions” or “behavioral economics”) of the past fifty years. We wrote up our ideas and, years later, our article made it into […]
Sequential Bayesian Designs for Rapid Learning in COVID-19 Clinical Trials
This from Frank Harrell looks important: This trial will adopt a Bayesian framework. Continuous learning from data and computation of probabilities that are directly applicable to decision making in the face of uncertainty are hallmarks of the Bayesian approach. Bayesian sequential designs are the simplest of flexible designs, and continuous learning capitalizes on their efficiency, […]
Can someone build a Bayesian tool that takes into account your symptoms and where you live to estimate your probability of having coronavirus?
Carl Mears writes: I’m married to a doctor who does primary care with a mostly disadvantaged patient base. The problem her patients face is if they get tested for COVID, they are supposed to self quarantine until they get their test results, which currently takes something like a week. Also, their *family* is supposed to […]
This one’s for the Lancet editorial board: A trolley problem for our times (involving a plate of delicious cookies and a steaming pile of poop)
A trolley problem for our times OK, I couldn’t quite frame this one as a trolley problem—maybe those of you who are more philosophically adept than I am can do this—so I set it up as a cookie problem? Here it is: Suppose someone was to knock on your office door and use some mix […]
This is not a post about remdesivir.
Someone pointed me to this post by a doctor named Daniel Hopkins on a site called KevinMD.com, expressing skepticism about a new study of remdesivir. I guess some work has been done following up on that trial on 18 monkeys. From the KevinMD post: On April 29th Anthony Fauci announced the National Institute of Allergy […]
covidcp.org: A COVID-19 collaboration platform.
Following up on today’s post on design of studies for coronavirus treatments and vaccines, Z points to this site, which states: In the U.S. only a few COVID-19 randomized clinical trials (RCTs) have been centrally organized, e.g. by NIAID, PCORI and individual PIs. Over 400 such trials have been registered on clinicaltrials.gov with dozens being […]
This one’s important: Designing clinical trials for coronavirus treatments and vaccines
I’ve had various thoughts regarding clinical trials for coronavirus treatments and vaccines, and then I came across thoughtful posts by Thomas Lumley and Joseph Delaney on vaccines. So let’s talk, first about treatments, then about vaccines. Clinical trials for treatments The first thing I want to say is that designing clinical trials is not just […]
Years of Life Lost due to coronavirus
This post is by Phil Price, not Andrew. A few days ago I posted some thoughts about the coronavirus response, one of which was that I wanted to see ‘years of life lost’ in addition to (or even instead of) ‘deaths’. Mendel pointed me to a source of data for Florida cases and deaths, which […]