Inspired by this effort by Patrick Collison for Silicon Valley [linked from Tyler Cowen], I thought that it might be fun to think about what makes up the Bayesian canon. As Collison said, “This isn’t the list of books that I think one ought to read — it’s just the list that I think roughly covers the major ideas that are influential here.”
I’ve made a start. Would should be added/removed?
Books:
- Berger, James O. (1985). Statistical Decision Theory and Bayesian Analysis. Springer.
- de Finetti, Bruno. (1974). Theory of Probability: A Critical Introductory Treatment. Wiley.
- Gelman, Andrew, and Jennifer Hill. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press.
- Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. (2013) [1995]. Bayesian Data Analysis. Chapman & Hall/CRC.
- Jaynes, E.T. (2003). Probability Theory: The Logic of Science. Cambridge University Press.
- Jeffreys, Harold. (1998) [1939]. Theory of Probability. Oxford University Press.
- McElreath, Richard. (2020) [2015]. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. CRC Press.
- Savage, Leonard J. [1974] (1954). The Foundations of Statistics. Wiley.
- Stigler, Stephen M. (1986). The History of Statistics: The Measurement of Uncertainty Before 1900. Belknap Press of Harvard University Press.
- Tukey, John. (1977). Exploratory Data Analysis. Addison-Wesley
Articles:
- Bayes, Thomas. (1763). “An Essay towards Solving a Problem in the Doctrine of Chances.” Philosophical Transactions of the Royal Society of London, 53, 370–418.
- Betancourt, Michael. (2017). “A Conceptual Introduction to Hamiltonian Monte Carlo.” arXiv. https://arxiv.org/abs/1701.02434.
- Fienberg, Stephen E. (2006). “When Did Bayesian Inference Become ‘Bayesian’?” Bayesian Analysis, 1(1), 1–37. https://doi.org/10.1214/06-BA101.
- Gelman, Andrew, Aki Vehtari, Daniel Simpson, Charles Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. (2020). “Bayesian Workflow.” arXiv. https://doi.org/10.48550/arXiv.2011.01808.
- Navarro, D. J. (2019). “Between the devil and the deep blue sea: Tensions between scientific judgement and statistical model selection.” Computational Brain and Behavior 2, 28–34, https://doi.org/10.1007/s42113-018-0019-z.
- Neal, Radford (2011). “MCMC Using Hamiltonian Dynamics”, in Handbook of Markov Chain Monte Carlo, CRC Press. (eds. Stephen Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng), https://www.mcmchandbook.net/HandbookChapter5.pdf.
- Raftery, Adrian E. (1995). “Bayesian Model Selection in Social Research.” Sociological Methodology, 25, 111–163, https://doi.org/10.2307/271063.
- Wang, Wei, David Rothschild, Sharad Goel, and Andrew Gelman. (2015). “Forecasting Elections with Non-Representative Polls.” International Journal of Forecasting, 31(3), 980–991. https://doi.org/10.1016/j.ijforecast.2014.06.001.
Blogs/online writing:
- Betancourt: https://betanalpha.github.io/writing/.
- brms vignettes: https://paulbuerkner.com/brms/articles/index.html.
- Data Colada: https://datacolada.org.
- Gelman: https://statmodeling.stat.columbia.edu.
- Heiss: https://www.andrewheiss.com/blog/.
- Kurz: https://solomonkurz.netlify.app/book/.
- rstanarm vignettes: https://mc-stan.org/rstanarm/articles/index.html.
- Stan Documentation: https://mc-stan.org/users/documentation/.
- Stan Forums: https://discourse.mc-stan.org.
Rohan:
Thanks for the list. It’s a great start, and I appreciate you including two of my books and two of my articles! Also, since you’re including blogs, I recommend those of Frank Harrell and Thomas Lumley.
And I have one more addition for you. Your list includes Raftery (1995), a paper that I strongly disagree with. If you’re gonna include that article, I also recommend my response to that article, [1995] Avoiding model selection in Bayesian social research. Sociological Methodology 1995, 165-173 (Andrew Gelman and Donald B. Rubin).
This last bit suggests another list we could prepare, which would be readings on various controversies in statistics.
And here’s a list of 10 statistics articles that Aki and I recommended as a companion piece to our recent review article, What are the most important statistical ideas of the past 50 years?.
Finally, I recommend a recent book you might have heard of called Telling Stories with Data.
Consider adding Blei’s JASA review of variational inference and Dunson’s article on Bayesian nonparametric hierarchical models.
In addition to those, I’d add the book on Bayesian nonparametric models by Ghosal and van der Vaart.
You need something by Lindley.
Understanding Uncertainty is great and would deserve to be better known – the high price doesn’t help though.
https://www.wiley.com/en-us/Understanding+Uncertainty%2C+Revised+Edition-p-9781118650127
I would add the book ‘Gaussian Processes for Machine Learning’ by Rasmussen and Williams. It’s really influential wrt Bayesian ideas in ML.
Jack Cohen’s brainchild,
not really Bayesian but a major breakthrough in the social sciences.
Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70(6, Pt.1), 426–443. https://doi.org/10.1037/h0026714
That Silicon Valley list explains a lot.
It’s a good list. I would add David McKay’s book on information theory as well (https://www.inference.org.uk/itprnn/book.pdf)
Great list. In medicine, the Spiegelhalter et al article “Bayesian approaches to randomized trials” has been tremendously impactful and certainly a big part of the Bayesian canon: https://www.jstor.org/stable/pdf/2983527.pdf
You can actually see in the discussion commentaries at the end of the pdf its influence on taking Bayes seriously, as recounted by Frank Harrell here: https://www.fharrell.com/post/journey/
MacKay needs to be in the list
+1 for MacKay
There should be a book on boating safety.
You might consider including some popular histories of Bayesian statistics like Sharon McGrayne’s “The Theory that Would Not Die” and Aubrey Clayton’s “Bernoulli’s Fallacy”.
This list might make a good contribution to the “awesome” project on GitHub: https://github.com/sindresorhus/awesome. Putting in the form of a git repository would also let you make use of the pull request system to review and integrate proposed additions.
Thanks for the Aubrey Clayton reference.
I was not aware of the book but I remember the name: https://statmodeling.stat.columbia.edu/2020/10/12/more-on-martingale-property-of-probabilistic-forecasts-and-some-other-issues-with-our-election-model/
(In unrelated election models news, FiveThirtyEight’s forecast is back.)
Carlos, shortly after that exchange (hard to believe it was 4 years ago already) I had a brief back and forth with Aubrey via email. he told me about his book which was in progress at the time. I got it and have recommended it to people who don’t have any background in Bayes to better understand the Frequentist vs Bayesian distinction and the kinds of concerns we discuss daily here. One friend who is in involved in the LA County public services realm has taken it strongly to heart and is considering taking a Bayesian stats class to better understand how to interpret studies. That’s a win I think. Thanks Aubrey and I fully agree it needs to be on the list.
Raiffa and Schleiffer Applied Statistical Theory should be considered. It was a pain to use without the index but everything you needed was there…somewhere.
Zellner, Arnold An Intro to Bayesian Inference in Econometrics.
Scientific Reasoning: The Bayesian Approach
Book by Colin Howson and Peter Urbach
Christian Robert’s blog (https://xianblog.wordpress.com/) deserves a mention as well. A more comprehensive recommended reading list (highly subjective of course) is given in Chapter 31 of my free course book with Dora Matzke (https://www.bayesianspectacles.org/free-course-book/).
One of the first entries in that chapter is this book, which is a great read: Howie, D. (2002). Interpreting Probability: Controversies and Developments in the Early Twentieth Century.
What about S. Watanabe’s book
“Mathematical theory of Bayesian statistics”
which covers Bayesian statistics for non regular, non identifiable models (such as Bayesian neural networks) and explains how to adapt many standard tools in that context?
I suggest adding Aubrey Clayton’s “Bernoulli’s Fallacy”. This is a strong and ,to me, convincing argument that much of statisitical inference today is wrong.
From the amazon review it sounds like thats about NHST. In that case the statistical inference isnt really wrong, just utterly worthless.
Then people jump to whatever conclusion they want based on this worthless (we know the correlation/difference is not exactly zero beforehand) inference.
Ie, the error is not in the stats per se. Its using it to check whether a strawman “coincidence” model explains the observations.
It’s not really about NHST it’s about Frequency vs Bayesian probability. He is basically popularizing the line of argument from ETJayes.
The title refers to the transposed conditional fallacy then? Ie, p(H|D) != p(D|H).
Its still proportional though:
p(H[0]|D) = C*p(D|H[0]), where
C = p(H[0])/sum(p(H[0:n])*p(D|H[0:n])
So lots of inferences (eg confidence intervals) can still be ok. According to the book, whats the nature of all the incorrect statistical inferences?
In the end it’s more of a book explaining the problems and how the different styles of statistics work. He discusses history and then in a chapter on the typical frequentist methodology he uses a fictional Chatbot called SuperFreq in a kind of Socratic dialog to explore the illogic of NHST and the ways it can go badly wrong. I don’t think you can summarize it with a simple pithy statement except maybe that doing Bayesian statistics thoughtfully will never violate cox’s theorems and protect you from many illogical outcomes.
Kruschke’s textbook was my first real introduction to Bayesian statistics. I’m still a novice, so I don’t know it rises to the level of “canon”, but it certainly helped me get my feet wet with Bayesian statistics.
Good to see Jaynes being recommended here. Three other books written by physical scientists deserve to be mentioned:
– ‘Bayesian Logical Data Analysis for the Physical Sciences’ by Gregory (2005),
– ‘Data Analysis: A Bayesian Tutorial’ by Sivia and Skilling (2006),
– ‘Practical Bayesian Inference: A Primer for Physical Scientists’ by Bailer-Jones (2017).
And I would like to add my own book, ‘Bayesian Compendium’ (2020, 2nd ed. 2024) to that list.
I am fond of Bernardo & Smith’s Bayesian Theory.