There’s a saying in negotiation that the most powerful asset is the ability to walk away from the deal. Similarly, in science (or engineering, business decision making, etc.), you have to be willing to give up your favorite ideas. When I look at various embarrassing examples in science during the past decade, a common thread […]
Archive of posts filed under the Decision Theory category.
Is “abandon statistical significance” like organically fed, free-range chicken?
The question: is good statistics scalable? This comes up a lot in discussions on abandoning statistical significance, null-hypothesis significance testing, p-value thresholding, etc. I recommend accepting uncertainty, but what if it’s decision time—what to do? How can the world function if the millions of scientific decisions currently made using statistical significance somehow have to be […]
Participate in Brazilian Reproducibility Initiative—even if you don’t live in South America!
Anna Dreber writes: There’s a big reproducibility initiative in Brazil on biomedical research led by Olavo Amaral and others, which is an awesome project where they are replicating 60 studies in Brazilian biomedical research. We (as usual lots of collaborators) are having a prediction survey and prediction markets for these replications – would it be […]
When Prediction Markets Fail
A few years ago, David Rothschild and I wrote: Prediction markets have a strong track record and people trust them. And that actually may be the problem right now. . . . a trader can buy a contract on an outcome, such as the Democratic nominee to win the 2016 presidential election, and it will […]
The dropout rate in his survey is over 60%. What should he do? I suggest MRP.
Alon Honig writes: I work for a cpg company that conducts longitudinal surveys for analysis of customer behavior. In particular they wanted to know how people are interacting with our product. Unfortunately the designers of these surveys put so many questions (100+) that the dropout rate (those that did not complete the survey) was over […]
Givewell is hiring; wants someone to help figure out how to give well; Bayesian methods may be relevant here
Josh Rosenberg writes: GiveWell (www.givewell.org) is a nonprofit that does in-depth research to direct funds to outstanding organizations helping the global poor. In 2018, we directed more than $140 million to our recommendations. We are recruiting researchers at varying levels of seniority to identify the giving opportunities which can most cost-effectively improve the lives of […]
Afternoon decision fatigue
Paul Alper points us to this op-ed, “Don’t Visit Your Doctor in the Afternoon,” by Jeffrey Linder: According to the study, published in JAMA Network Open, doctors ordered fewer breast and colon cancer screenings for patients later in the day, compared to first thing in the morning. All the patients were due for screening, but […]
“Causal Processes in Psychology Are Heterogeneous”
Niall Bolger sends along this article he wrote with Katherine Zee, Maya Rossignac-Milon, and Ran Hassin, which begins: All experimenters know that human and animal subjects do not respond uniformly to experimental treatments. Yet theories and findings in experimental psychology either ignore this causal effect heterogeneity or treat it as uninteresting error. This is the […]
We’re hiring an econ postdoc!
It’s for hierarchical modeling for policy analysis in Stan. We’re really excited about this project. Will share more details soon, but wanted to get this out right away.
Bayesian analysis of data collected sequentially: it’s easy, just include as predictors in the model any variables that go into the stopping rule.
Mark Palko writes: I remember you did something on the practice of continuing to add to the sample until significance was reached. I wanted to share it with some co-workers but I can’t seem to find it on your blog. Do you remember the one I am talking about? My reply: It’s here. There’s more […]
When presenting a new method, talk about its failure modes.
A coauthor writes: I really like the paper [we are writing] as it is. My only criticism of it perhaps would be that we present this great new method and discuss all of its merits, but we do not really discuss when it fails / what its downsides are. Are there any cases where the […]
The best is the enemy of the good. It is also the enemy of the not so good.
This post is by Phil Price, not Andrew. The Ocean Cleanup Project’s device to clean up plastic from the Great Pacific Garbage Patch is back in the news because it is back at work and is successfully collecting plastic. A bunch of my friends are pretty happy about it and have said so on social […]
What’s the p-value good for: I answer some questions.
Martin King writes: For a couple of decades (from about 1988 to 2006) I was employed as a support statistician, and became very interested in the p-value issue; hence my interest in your contribution to this debate. (I am not familiar with the p-value ‘reconciliation’ literature, as published after about 2005.) I would hugely appreciate […]
Automation and judgment, from the rational animal to the irrational machine
Virgil Kurkjian writes: I was recently going through some of your recent blog posts and came across Using numbers to replace judgment. I recently wrote something about legible signaling which I think helps shed some light on exactly what causes the bureaucratization of science and maybe what we can do about it. In short I […]
Glenn Shafer: “The Language of Betting as a Strategy for Statistical and Scientific Communication”
Glenn Shafer writes: I have joined the immense crowd writing about p-values. My proposal is to replace them with betting outcomes: the factor by which a bet against the hypothesis multiplies the money it risks. This addresses the desideratum you and Carlin identify: embrace all the uncertainty. No one will forget that the outcome of […]
My talk at the Brookings Institution this Fri 11am
The replication crisis in science: Does it matter for policy? Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University I argue that policy analysts should care about the replication crisis for three reasons: (1) High-profile policy claims have been systematically exaggerated; (2) This has implications for how to conduct and interpret new […]
Are statistical nitpickers (e.g., Kaiser Fung and me) getting the way of progress or even serving the forces of evil?
As Ira Glass says, today we have a theme and some variations on this theme. Statistical nitpickers: Do they cause more harm than good? I’d like to think we cause more good than harm, but today I want to consider the counter-argument, that, even when we are correct on the technical merits, we statisticians should […]
The Map Is Not The Territory
This post is by Phil Price, not Andrew. My wife and I are building a new house, or, rather, paying trained professionals to build one for us. We are trying to make the house as environmentally benign as we reasonably can: ducted mini-split heating, heat pump water heater, solar panels, heat-recovery ventilator, sustainably harvested lumber, […]
Econ corner: A rational reason (beyond the usual “risk aversion” or concave utility function) for wanting to minimize future uncertainty in a decision-making setting
Eric Rasmusen sends along a paper, Option Learning as a Reason for Firms to Be Averse to Idiosyncratic Risk, and writes: It tries to distinguish between two kinds of risk. The distinction is between uncertainty that the firm will learn about, and uncertainty that will be bumping the profit process around forever. It’s not the […]
Glenn Shafer tells us about the origins of “statistical significance”
Shafer writes: It turns out that Francis Edgeworth, who introduced “significant” in statistics, and Karl Pearson, who popularized it in statistics, used it differently than we do. For Edgeworth and Pearson, “being significant” meant “signifying”. An observed difference was significant if it signified a real difference, and you needed a very small p-value to be […]