A couple years ago, Jay Naborn wrote:
I am studying people’s preference for categorically correct forecasts (such as getting the winner of a sports game right) over error-minimizing ones (such as getting close on the margin). We have experimental evidence of this, why it happens, etc.
What I would be interested in doing is demonstrating that this preference is/can be a mistake. To do so, it would be nice to show that doing well in terms of minimizing continuous error is a better predictor of future winner-picking than is doing well in terms of winner-picking. I am curious if you have any leads as to some existing dataset that would be helpful here, or some simulation/modeling strategy that may work.
I replied that, yes, this relates to a point we made here.
Recently Naborn followed up:
The blog post you sent (and a couple others of yours) were very informative for our background thinking. My work (with Jonathan Bogard) forecast evaluation is now published at the Journal of Marketing Research.
And here’s the abstract:
People routinely make decisions based on predictions made by others (e.g., political pundits, market analysts), so it is in their best interest to identify high-quality forecasts. Experts characterize good forecasting as minimization of continuous error (i.e., predictions close to the eventual outcome). By contrast, the present work reveals that laypeople typically see good forecasts as those that correctly predict an event’s categorical outcome (e.g., the winning team). Using within-subjects, between-subjects, and incentive-compatible designs, fifteen studies demonstrate this “pick-the-winner-picker heuristic” as well as its psychological mechanism: People evaluate forecasts by assigning separate weights to (a) categorical correctness and (b) continuous error minimization, depending on the overall importance of the categorical and continuous dimensions for that situation. Thus, in the common case when the categorical dimension matters most (e.g., sports contests), people prize forecasts that accurately predicted the categorical outcome (e.g., the winner, not the margin of victory). However, when the categorical dimension’s stakes are experimentally reduced, an attenuation is observed. While this describes how people typically evaluate forecasts, crucially, a dimension’s importance is not necessarily related to its diagnosticity of forecaster skill or reliability. Accordingly, the pick-the-winner-picker heuristic may constitute a normative mistake, while framing manipulations help debias judgments.
Interesting. It’s good to see research on this topic.
I am browsing the pre-print or post-print posted on Researchgate. The following is a quote concerning experiment 5 titled “Chance Event Study”:
“They were told that two weeks before Election Day, the weather forecast indicated a 60% chance of rain in the part of the district that strongly favored Winters and where most of Winters’ supporters live. They were then told that many of the people in this district would stay home and not vote if it ended up raining. Participants were then shown two forecasts made two weeks before the election: the winner-picker predicted Winters would win by 9%, the error-minimizer predicted Lester would win by 1%, and the actual outcome of the election was that Winters won by 2%. Importantly, participants were told that no rain fell on Election Day. The categorical dimension of the election outcome (which candidate won) was therefore determined, or at least strongly influenced, by the chance event of rain having not fallen. In other words, it is a purely aleatory event that caused the winner-picker to have ended up picking the winner.”
Uhm, I don’t understand perhaps but I am missing information about whether or not the forecasters included the specific weather predications in their forecasts. It could be that I am not understanding correctly, but it seems to me that that information is not presented to the participants, nor is it clear whether the forecasters used this weather forecast in their predictions. If this is correct, how can it be concluded by the authors that:
“Complementing our theoretical concerns that picking the winner-picker may involve rewarding luck, participants in Study 5 rated a winner-picker as better than an error-minimize even when no skill could have been involved in ultimately predicting the correct categorical outcome.”
To me it’s always so odd to say that people are making mistakes rather than our model does not match how people think about things.
Partly as a result of browsing Naborn & Bogard (2025)’s “Pick-the-Winner”-heuristic paper, I would like to hereby propose a possible new heuristic: the “Social Scientist Heuristic and Bias”-heuristic and bias. The Social Scientist Heuristic and Bias-heuristic and bias is a phenomenon where social scientists look at some reasoning or decision or judgment process and are inclined to think about this all in terms of some bias or heuristic or even some mistake. This possible phenomenon became more clear to me after reading certain sections of Naborn & Bogard (2025).
In certain cases the Social Scientist Heuristic and Bias- heuristic and bias may lead to social scientists going to great lengths to interpret and view some reasoning or decision making or judgment process as being a “heuristic” or as “biased” (whatever these terms exactly mean or imply). For instance, some social scientists might write that: “More specifically, we propose a “contingent-weighting hypothesis”: People evaluate forecasts by assigning separate weights to (a) categorical correctness and (b) continuous error minimization, depending on the overall importance of the categorical and continuous dimensions (for that situation).”. And some social scientists might write that: “Importantly, this general tendency may typically appear as a preference for winner-picking forecasts since the categorical dimension of an outcome is often what matters most (who won the election, whether a project funding goal was reached, and so on).” Still, given these sentences, and the possible interpretation that this process seems to involve pretty complex and nuanced reasoning and decision making, and is even often most important and most useful, the Social Scientist Heuristic and Bias-heuristic and bias may still, almost implicitly, result in social scientists looking for some reason why this reasoning and decision process might be “flawed” or “irrational” or can be called a “heuristic” or a “bias” or even a “mistake”.
Further evidence for the Social Scientist Heuristic and Bias-heuristic and bias can perhaps also be found in the mere words and terms used in certain research. For instance, a perhaps more objective and appropriate use of terms like “categorical dimension preference” and “continuous dimension preference” are not used, but instead terms like “winner-pickers” and “error-minimizers” are used instead. Such terms may already point to a certain interpretation and conclusion of findings and the process as a whole, possibly influenced by the Social Scienctist Heuristic and Bias-heuristic and bias.
The Social Scientist Heuristic and Bias-heuristic and bias can possibly be present in very subtle ways. For instance, the sentence “Given all of the above, we hypothesize that laypeople will often prefer forecasts that are categorically correct to those that are closer to the eventual outcome.” may subtly show the bias by connecting and associating “laypeople” to the categorically correct preference. This is done whilst also noting that “Despite all this, if the categorical outcome is the only dimension that people care about, it is reasonable to wonder whether the pick-the-winner-picker heuristic may be a sensible strategy.”. Even though it can be seen as a sensible strategy, possiby used by laypeople, in subsequent discussions however socials scientists might state: “That experts, relative to laypeople, are less drawn to winner-pickers further suggests that using the pick-the-winner-picker heuristic may be a mistake.”. There could perhaps be many reasons why “experts” gave different answers, and showed no special privilege to categorical correctness, in a certain specific experiment, but the Social Science Heuristic and Bias- heuristic and bias may result in social scientists almost automatically reason and assume this is due to the “experts” being less wrong, or reasoning “better”, or having less of a “bias”.
Finally, in certain cases the Social Scientist Heuristic and Bias-heuristic and bias may lead to designing and reporting an experiment by social scientists in which a certain heuristic might become apparent. This might sometimes be done in such a way however that it might show more of the Social Scientist Heuristic and Bias-heuristic and bias than some other heuristic like the Pick-the-Winner heuristic. For example, a certain comment on a certain statistical modeling blog shows that a closer look at experiment 5 by Naborn & Bogard (2025) might provide some more information and clarity about this possibility. Regardless of the specifics of experiment 5 of Naborn and Bogard (2025), and whether or not it might indicate the heuristic or bias, it might be interesting for further research to ponder possible origins, or reasons, concerning the Social Scientist Heuristic and Bias-heuristic and bias. Are there social scientists who like to come up with some uniquely-named heuristic to then attempt to make a career out of it and have their name associated with the heuristic? Might this be a way for social scientists to go on TV, or be interviewed by some newspaper, and talk about their specific uniquely-named heuristic when it might be relevant to do so, for instance when the heuristic might be connected to some event in the news like a new election? Or, could it be a way for social scientists to label some things a certain way because that might be useful for some other thing(s)? Further discussion and research may shed some more light on the Social Scientists Heuristic and Bias- heuristic and bias in multiple ways. For now, this has merely been an introductory discussion about a phenomenon that might be noteworthy and might deserve some more attention.
Yeah. I think that has sort of been Gigerenzer’s meta point all along. Maybe not so odd knowing what we know about supply and demand for scientific discoveries, realities of signal and noise in human cognition and behavior, conventions around communicating results, and the pressures to hire young researchers who can deliver great storylines.
I am the Anonymous in the following link to a comment regarding Gigerenzer and what you might be refering to in your comment here:
https://statmodeling.stat.columbia.edu/2019/07/14/gigerenzer-the-bias-bias-in-behavioral-economics-including-discussion-of-political-implications/#comment-1081030