Bill Harris points to this press release, “Almost 400 medical practices found ineffective in analysis of 3,000 studies,” and asks: The intent seems good; does the process seem good, too? For one thing, there is patient variation, and RCTs seem focused on medians or means. Right tails can be significant. This seems related to the […]
Archive of posts filed under the Decision Theory category.
“Pfizer had clues its blockbuster drug could prevent Alzheimer’s. Why didn’t it tell the world?”
Jon Baron points to this news article by Christopher Rowland: Pfizer had clues its blockbuster drug could prevent Alzheimer’s. Why didn’t it tell the world? A team of researchers inside Pfizer made a startling find in 2015: The company’s blockbuster rheumatoid arthritis therapy Enbrel, a powerful anti-inflammatory drug, appeared to reduce the risk of Alzheimer’s […]
Unquestionable Research Practices
Hi! (This is Dan.) The glorious Josh Loftus from NYU just asked the following question. Obviously he’s not heard of preregistration. Seriously though, it’s always good to remember that a lot of ink being spilled over hypothesis testing and it’s statistical brethren doesn’t mean that if we fix that we’ll fix anything. It all comes to […]
In short, adding more animals to your experiment is fine. The problem is in using statistical significance to make decisions about what to conclude from your data.
Denis Jabaudon writes: I was thinking that perhaps you could help me with the following “paradox?” that I often find myself in when discussing with students (I am a basic neuroscientist and my unit of counting is usually cells or animals): When performing a “pilot” study on say 5 animals, and finding an “almost significant” […]
The default prior for logistic regression coefficients in Scikit-learn
Someone pointed me to this post by W. D., reporting that, in Python’s popular Scikit-learn package, the default prior for logistic regression coefficients is normal(0,1)—or, as W. D. puts it, L2 penalization with a lambda of 1. In the post, W. D. makes three arguments. I agree with two of them. 1. I agree with […]
“Machine Learning Under a Modern Optimization Lens” Under a Bayesian Lens
I (Yuling) read this new book Machine Learning Under a Modern Optimization Lens (by Dimitris Bertsimas and Jack Dunn) after I grabbed it from Andrew’s desk. Apparently machine learning is now such a wide-ranging area that we have to access it through some sub-manifold so as to evade dimension curse, and it is the same […]
“Whether something is statistically significant is itself a very random feature of data, so in this case you’re essentially outsourcing your modeling decision to a random number”
I happened to come across a post of mine that’s not scheduled until next April, and I noticed the above line, which I really liked, so I’m sharing it with you right now here. The comment relates to a common procedure in statistics, where researchers decide exclude potentially important interactions from their models, just because […]
No, Bayes does not like Mayor Pete. (Pitfalls of using implied betting market odds to estimate electability.)
Asher Meir points to this amusing post from Greg Mankiw, who writes: Who has the best chance of beating Donald Trump? A clue can be found using Bayes Theorem. Here is the logic. Let A be the event that a candidate wins the general election, and B be the event that a candidate wins his […]
What does a “statistically significant difference in mortality rates” mean when you’re trying to decide where to send your kid for heart surgery?
Keith Turner writes: I am not sure if you caught the big story in the New York Times last week about UNC’s pediatric heart surgery program, but part of the story made me interested to know if you had thoughts: Doctors were told that the [mortality] rate had improved in recent years, but the program […]
What’s the evidence on the effectiveness of psychotherapy?
Kyle Dirck points us to this article by John Sakaluk, Robyn Kilshaw, Alexander Williams, and Kathleen Rhyner in the Journal of Abnormal Psychology, which begins: Empirically supported treatments (or therapies; ESTs) are the gold standard in therapeutic interventions for psychopathology. Based on a set of methodological and statistical criteria, the APA [American Psychological Association] has […]
In research as in negotiation: Be willing to walk away, don’t paint yourself into a corner, leave no hostages to fortune
There’s a saying in negotiation that the most powerful asset is the ability to walk away from the deal. Similarly, in science (or engineering, business decision making, etc.), you have to be willing to give up your favorite ideas. When I look at various embarrassing examples in science during the past decade, a common thread […]
Is “abandon statistical significance” like organically fed, free-range chicken?
The question: is good statistics scalable? This comes up a lot in discussions on abandoning statistical significance, null-hypothesis significance testing, p-value thresholding, etc. I recommend accepting uncertainty, but what if it’s decision time—what to do? How can the world function if the millions of scientific decisions currently made using statistical significance somehow have to be […]
Participate in Brazilian Reproducibility Initiative—even if you don’t live in South America!
Anna Dreber writes: There’s a big reproducibility initiative in Brazil on biomedical research led by Olavo Amaral and others, which is an awesome project where they are replicating 60 studies in Brazilian biomedical research. We (as usual lots of collaborators) are having a prediction survey and prediction markets for these replications – would it be […]
When Prediction Markets Fail
A few years ago, David Rothschild and I wrote: Prediction markets have a strong track record and people trust them. And that actually may be the problem right now. . . . a trader can buy a contract on an outcome, such as the Democratic nominee to win the 2016 presidential election, and it will […]
The dropout rate in his survey is over 60%. What should he do? I suggest MRP.
Alon Honig writes: I work for a cpg company that conducts longitudinal surveys for analysis of customer behavior. In particular they wanted to know how people are interacting with our product. Unfortunately the designers of these surveys put so many questions (100+) that the dropout rate (those that did not complete the survey) was over […]
Givewell is hiring; wants someone to help figure out how to give well; Bayesian methods may be relevant here
Josh Rosenberg writes: GiveWell (www.givewell.org) is a nonprofit that does in-depth research to direct funds to outstanding organizations helping the global poor. In 2018, we directed more than $140 million to our recommendations. We are recruiting researchers at varying levels of seniority to identify the giving opportunities which can most cost-effectively improve the lives of […]
Afternoon decision fatigue
Paul Alper points us to this op-ed, “Don’t Visit Your Doctor in the Afternoon,” by Jeffrey Linder: According to the study, published in JAMA Network Open, doctors ordered fewer breast and colon cancer screenings for patients later in the day, compared to first thing in the morning. All the patients were due for screening, but […]
“Causal Processes in Psychology Are Heterogeneous”
Niall Bolger sends along this article he wrote with Katherine Zee, Maya Rossignac-Milon, and Ran Hassin, which begins: All experimenters know that human and animal subjects do not respond uniformly to experimental treatments. Yet theories and findings in experimental psychology either ignore this causal effect heterogeneity or treat it as uninteresting error. This is the […]
We’re hiring an econ postdoc!
It’s for hierarchical modeling for policy analysis in Stan. We’re really excited about this project. Will share more details soon, but wanted to get this out right away.
Bayesian analysis of data collected sequentially: it’s easy, just include as predictors in the model any variables that go into the stopping rule.
Mark Palko writes: I remember you did something on the practice of continuing to add to the sample until significance was reached. I wanted to share it with some co-workers but I can’t seem to find it on your blog. Do you remember the one I am talking about? My reply: It’s here. There’s more […]