Skip to content

News flash: Probability and statistics are hard to understand

Two people pointed me to an article by Emre Soyer and Robin Hogarth that was linked to by Felix Salmon.

Here are my reactions:

1. Soyer and Hogarth’s paper seems very strong to me, and Salmon’s presentation is an impressive condensation of it. I’d say good job on the science and the reporting.

2. I don’t see the point of focusing on economists. This seems just like a gimmick to me. But, then again, I’m not an economist. So of course I’d be more interested in a similar paper studying political scientists or statisticians. This should be easy enough for someone to do, of course.

3. To elaborate on this last point: I’m not surprised that people, even expert practitioners, screw up with statistics. Kahneman and Tversky found this with psychology researchers back in the 1970s. I’m not knocking the current paper by Soyer and Hogarth but I don’t see it as surprising. Perhaps the focus on economists is what allowed it to get all this attention. If you want people to read your newspaper article, write it about celebrities. If you want people to read your academic article, write it about economists?

4. Soyer and Hogarth’s paper is all about how difficult it is to understand statistical results presented as tables. I was disappointed (but, unfortunately, not surprised) to see them present many of their findings in tables rather than graphs, and the graphs they do use are uninspired—they’re not the worst graphs in the world, but they’re a bunch of poorly-organized bar charts. A collaboration with someone such as Kaiser Fung would’ve made the presentation a lot better (if they were to follow the minimal strategy of letting Kaiser improve, compress, and combine their graphs) or a lot lot lot lot better (if they were to think hard about presenting all their key results graphically). I would prefer to see all the numbers in Tables 1-3 and Appendix C presented graphically. As I’ve discussed on numerous occasions, such plots can table up less space as well as displaying relevant comparisons more effectively than the corresponding tables.

5. Going forward, I see the real question as how to better understand and communicate statistical results. To me the recommendation from the present paper is not so much to display regressions as graphs (although I agree with this advice) but rather to use the statistical model to answer any questions of interest (what we call qoi’s) directly. For example, if you want people to know the value of x for which Pr(y>0)=.95, just calculate it directly from the model. All the time I see talks where people present regressions and start interpreting the coefficients and making various indirect claims that could be answered from the model directly.

I think Soyer, Hogarth, and Salmon for giving us the opportunity to think about all this and to bang home again once more the message we’ve learned from so much of the work of Kahneman, Tversky, Gigerenzer, and others that uncertainty and variation are hard to understand, that people are overconfident about their guesses, and that how a problem is framed can make a difference in how people perform on a psychological task.


  1. C. Zorn says:

    I agree on most counts with what you say here. But I do wonder: Given that this paper — or at least Salmon’s summary of it — have now been seen/read by nearly everyone I know (thank you Twitter, Facebook, blogs, etc.), would we be able to replicate it with political scientists, statisticians, and so forth and get similar results. I know when I read the summary that it became clear to me how one could do what many of the economists were doing, and I made a mental note to be sure not to make that mistake myself.

  2. BN says:

    [Your link to Felix’s article is missing a leading h.]

    My concern (as I posted on Felix’s website) is the following:

    Most economists’ studies that I’ve seen are investigating the marginal effect of X on Y. One such example might be an examination of education (X) on net worth (Y). Putting aside the very serious econometric issues (e.g., self-selection), those economists focus on the estimated coefficient on X. That’s because they care about whether education influences net worth (or income, or welfare, or whatever).

    Sure, it would be nice if they were also able to answer the question posed by Felix and the Soyer/Hogarth, too, but it’s an entirely different question: At what level of education are you 95% likely to observe a random individual has a positive net worth? That question has as much to do with variation in baseline net worth (having nothing to do with education) as it does with the link between education and net worth.

    That doesn’t seem like the type of question that economists ever ask. So on the one hand, yes, the Soyer/Hogarth results suggest that economists may not have as complete an understanding of statistics as we’d wish. But I disagree with the Salmon’s conclusion. He says
    “And so it’s easy to see, I think, how economists become convinced of things that the rest of us aren’t sure of at all — and how the economists often end up being wrong, while the rest of us were right to be dubious.”

    But nothing in the Soyer/Hogarth example (at least as summarized by Salmon) indicates that the economists are wrong *about the questions they’re asking*. To sum it up, if I care about the effect of education on income/net worth, I’m very much concerned about careful measurement and identification in order to estimate the marginal effect. And that marginal effect may differ across different groups of people, which is interesting, too. And if I can do all that carefully, I’m not sure why it’s a big deal that I get tripped up on an entirely unrelated question like, “At what level of education am I 95% likely to observe a positive net worth?”. Especially when all of the noise in the Salmon example is coming from the estimate of the constant term, rather than from the marginal effect.

  3. Dan says:

    Several of the comments on that blog are complaining about the wording of the question. They claim the failure to answer correctly was due to poor phrasing. The problem is that a person’s level of statistical knowledge will directly relate to their probability of correctly interpreting the question. So claiming that you couldn’t “understand” the question doesn’t let you off the hook – it’s direct proof of the original message!

    Also direct proof: arguing that the question should’ve asked for the minimum value of X.

    • jrkrideau says:

      And from the original paper
      For Conditions 1, 3, and 5, we asked the following questions:
      1. What would be the minimum value of X that an individual would need to make sure that s/heobtains a positive outcome (Y > 0) with 95% probability?

      It seems pretty clear although I didn’t have any problem understanding the question on the blob.

      For the record, just eyeballing the graph I went for somewere between 51 & 54. Duh!

  4. John says:

    Is the “right” answer provided in the paper correct? A better answer would also take into account the uncertainty in the betas, not only the uncertainty in the error term. In other words, using posterior predictive checks as in your book with Hill, chapter 7.

  5. Ned says:

    Error in the link to Felix Salmon:

    Needs an initial “h”

    Nice post by Salmon.

  6. Andrew, the second link in the first paragraph (“linked to by Felix Salmon”) doesn’t work; it should have been the first link, and the first link should have been the one to the Soyer/Hogarth article, which is this link:

  7. zbicyclist says:

    1. Economists are “hot” now (Freakonomics effect?)
    2. Economists are presumably following a rational, calculational model and can be expected to know statistics. I’m not sure that’s true of political scientists or psychologists as a class.
    3. Despite #1 and #2, people are skeptical about economics and economists. This may have something to do with the economy.

  8. ezra abrams says:

    actually, their original graphs (figs 3 and 4) arn’t that good either, are they ? I mean, tukey era people would not be happy
    just two points:
    you should scale your Y axis so the data make a 45o angle, roughly, this is where human vision/brain does best
    you should scale your X axis so that points don’t over lap the Y, esp if X vales are all >0
    not to mention, this wierd econ thing about putting the legend on a seperate page…like WRONG !!!

  9. emre says:


    We are Emre and Robin, the authors of the study featured in this post.

    We would like thank Andrew Gelman for his insightful discussion of the study and all the commentators for their remarks.

    This study is now a discussion paper in International Journal of Forecasting. Hence, we would like to contribute to the ongoing discussion by posting links to the comments made on the study and our reply to those comments.

    Due to copyright issues, we cannot share freely the journal versions, but can put links to the last working papers.

    Here is the study:

    Here is the comment paper by Scott Armstrong:

    Here is the comment paper by Stephen Ziliak:

    Here is the comment paper by Nassim Taleb and Daniel Goldstein:

    Here is the comment paper by Keith Ord:

    Here is our reply to the comments:

    Thank you once more for the engaging discussion.

    Best wishes,

    Emre and Robin

  10. Brennan says:

    Not so sure that the study doesn’t simply prove that you can add lots of word salad to a simple idea and confuse people. The graph is easy because the full range of a data set is about 2sigma. So, when you see that, roughly, all data in range is above zero, you can guess the estimated value at 95% confidence directly.

    It is right there.

    The crazy overspecification of the simple plot made it harder, and I think intentionally so. If you had said: it is a plot. Slope is .32, sigma is 28, then the answer is clear (.32*x+56=0). Instead, lots of meaningless information was used to hide the simple relationship.

    Just plot the +-95% CI on the chart! THey are obvious by eye, and that would anser the question.

    I am not at all sure this study proves anything but the power of obsfucation.

  11. […] Felix Salmon on how economists get tripped up by statistics , a summary of The illustion of predictability: how regression statistics mislead experts by Emre Soyer and Robin Hogarth. Here’s Andrew Gelman’s response. […]

  12. […] News flash: Probability and statistics are hard to understand ( […]