Our wonderful new Bayesian Workflow book is now available as a free pdf! Just go the link—it’s right there!
I recommend getting the hard copy too because you’ll want to be able to read it while working on the computer, and the cost of the book is trivial compared to the benefit from faster learning that you will get by being able look at the book without taking up valuable screen real estate; also you can see connections when flipping through the pages that might not be apparent by viewing one page at a time on a screen.
Conversely, if you have the hard copy, you should still download the pdf because it fixes a bunch of minor errors that we caught after the book went to press. Also in the printed version we accidentally repeated some of the exercises in chapters 2 and 3. For the pdf we fixed this.
Regarding the content, as I wrote last month, with Bayesian Data Analysis, the big steps forward were:
- Going beyond Bayesian inference to also consider Bayesian model building (as a researcher, you construct the model, it isn’t just given to you as in a textbook), model checking (breaking through the absolutely horrible attitude, common to Bayesians in the early 1990s, that the model was “subjective” and thus should not be checked), and model improvement (continuous model expansion, not the misguided idea of assigning posterior probabilities).
- Going beyond simple conjugate models. BDA had lots of hierarchical models, also lots of computational tools so that you could fit the models you want by putting them together from understandable components. And I like how we had a clear separation between modeling and computing. The model comes first, then you figure out how to compute it. Or you set up a model that works within your computational constraints.
- A Bayesian approach to sampling and causal inference. This was Rubin’s framework in which unobserved units in the population and unobserved causal outcomes are treated as missing data and are part of a joint probability model. We worked this out in chapter 7 of BDA (which became chapter 8 in the third edition of the book).
- Lots of live examples. Not just “real-data examples,” but problems we’d directly worked on. This motivated us and I think it gave our readers a sense of how Bayesian methods worked not just in theory but in applied problems.
- A pragmatic view of probability as a measurable quantity. That’s right there in chapter 1. Bayesian methods are not the product of a philosophical stance; they’re a way to connect models and data using probability.
I could go on and on, but for that I can refer you to the Bayesian Data Analysis book.
And these are the key innovations of Bayesian Workflow:
- Going beyond Bayesian data analysis (model building, inference, model checking, and model expansion) to consider the larger process of statistical modeling, including comparisons of multiple models fit to a single dataset.
- A fuller use of informative priors. This is a big deal. In BDA we still had a bit of the Bayesian cringe going on. One reason we’ve moved toward stronger priors is that the replication crisis has taught us that the amount of prior information available in any given problem is often approximately the same as the information coming from an experiment (see here, for example). Informative priors also fit our increased focus on generative modeling, and we’re doing a lot more prior predictive checking to understand the implications of our models.
- More integration between modeling, data analysis, and computing. One way to see this is that the Bayesian Workflow webpage has the code to run all our examples. We also have lots of code snippets in the text as a way of demonstrating the way in which coding is central to our statistical workflow.
- Lots more live examples. It’s been 30 years since BDA first came out. One reason that Bayesian Workflow has 11 authors is that different collaborators worked on different examples (but the three principal authors read through the entire book, so the general approach should remain coherent).
- Simulation-based experimentation. This is something my colleagues have been doing more and more over the years. At its most basic, simulation-based experimentation provides a best-case baseline for statistical methods: if you can’t recover your quantities of interest with sufficient accuracy under ideal conditions (when your data are simulated from the model you’re fitting), then you know you’re in trouble. And often this is the case! Beyond that, we can simulate from one model and fit another, and see what happens. Simulation experiments aren’t always so easy to construct, as they involve specifying the entire data-generation process. But we think this is effort worth expending, as it involves thinking about the problem you’re working on.
I could go on and on, but for that I can refer you to the Bayesian Workflow book.
Dear Prof Gelman, as a very long-standing follower of your blog and esteemer of (and learner from) your published work, I’d like to wholeheartedly thank you and your co-authors — and, as I’m at it, your numerous academic colleagues likewise believing in open, non-for-profit knowledge dissemination — for the generosity in making available your textbook(s) for non-commercial purposes. I very much appreciate and commend the gesture!
Stefano:
Thanks for the kind words. In turn, I will thank the public and private education systems in my country and in those of my collaborators for giving us the time and resources to do this work which we can now share with others.
Andrew –
Kinda related (and another reason why you merit respect):
However, I also made two observations that I found quite strange, if not downright unethical:
Academics frequently ask for money in deceptive ways, suggesting that paid subscriptions are “necessary,” “supportive,” “allow them to,” “are helpful,” in order to do public philosophy, despite being in secure tenured positions.
Academics frequently paywall their “public writing.”
https://walterveit.substack.com/p/is-it-unethical-for-academics-to?r=102wz&utm_medium=ios
Great work, and good OA promotion! Yet, I wonder: would you dare to share details about your contracts with CRC and other presses? I recently wrote a book chapter for a colleague’s edited volume and had to sign for a 48 month (!) embargo period.
Thank for making the book available.
I was surprised by the first paragraph I read (Why Bayes?): “All statistical frameworks provide a way to go from some data 𝑦 to inference about variables 𝜃. [Some statistical frameworks won’t consider those variable but fine.] But the Bayesian framework lets us just as easily go from 𝜃 to 𝑦, and this is useful in model development, model checking, and generalization. [What? You can always easily go from parameters to data. What’s special about the Bayesian framework is that it lets us go in the inverse direction!]
Fortunately the following paragraph looks more reasonable: “A classical statistical model is characterized by a probability distribution for data 𝑦given some vector of unknown parameters 𝜃…”
Carlos:
There are lots of non-Bayesian methods so it is even more impossible to generalize about them than to generalize about Bayesian methods!
But, no, you can’t “always easily go from parameters to data”! Maximum likelihood estimation depends on the likelihood, which is not the same as a generative model. And then there are methods such as quasi-likelihood, generalized estimating equations, second-order inference, etc., which make a virtue of not specifying a probability model for data. Which is fine—there can be good reasons for wanting an estimation method that is not tied to a particular probability model—but these are methods where there is no model for y given theta. I agree, though, that there also are non-Bayesian methods that specify a probability model for y given theta.
In any case, I agree with you that this string of sentences that we wrote can lead to confusion. I will think about how to rewrite this! So thanks for pointing it out.
> you can’t “always easily go from parameters to data”
You’re right. Initially I was going to specify “with probabilistic models” but I decided to keep that out for simplicity because at least in the context of your following paragraph it seems clear that for a classical statistical model you can easily go from parameters to data if it’s “characterised by a probability distribution for data 𝑦 given some vector of unknown parameters 𝜃”. And even though the parameters are not probabilistic that seems enough of a generative model to let you write recently that “Posterior predictive checking is for non-Bayesians too!”
(“All statistical frameworks provide a way to go from some data 𝑦 to inference about variables 𝜃” is also debatable. However, I guess that for non-parametric methods one may always come up with some 𝜃 to get some inference about as well. For a permutation test we could say that 𝜃 is 1 if the distributions are identical and 0 if they are not, for example.)
Carlos:
In practice it’s a big deal. One thing I like to say is that fitting a model is like playing SimCity, whereas conducting a fake-data simulation is like writing SimCity. Setting up a fake-data simulation requires making lots of additional choices beyond what’s in the likelihood. This is an issue even for Bayesians, because lots of Bayesian inference uses flat or very weak priors, but that won’t do for fake-data simulation—there, you need to take the prior more seriously as a generative model.
> Setting up a fake-data simulation requires making lots of additional choices beyond what’s in the likelihood.
So you can take some estimate for 𝜃 and simulate fake-data using the distribution for data 𝑦 given that 𝜃 and call that “posterior predictive checking” or not?
Carlos:
Yes, that’s ok if theta is estimated precisely in the fitted model. It’s what Mosteller and Bush did with the dogs experiment in their 1954 book, and that example was one of the things that inspired me to formalize the idea of posterior predictive checking.
But if theta is estimated with a lot of uncertainty, then it’s important for this uncertainty to be included in the posterior predictive distribution, and then using a point estimate for theta is not so great.
We discuss this in section A.9 of Bayesian Workflow . . . you can download the free pdf and take a look!
Thanks for the explanations but I have to say I remain confused. (If using a point estimate for the parameter is not great your “Posterior predictive checking is for non-Bayesians too!” post comes across as too celebratory. Those non-Bayesians – and you, by extension – are just slapping the “posterior” label on plug-in simulations.)
Anyway, thanks again for making the book freely available. (I had taken a look already, that’s how I was able to comment on it!)
Carlos:
Posterior predictive checking using the point estimate isn’t perfect but I think it’s a good start, and then there are natural next steps for simulating from some approximate uncertainty distribution of the parameter.
> Posterior predictive checking using the point estimate isn’t perfect but I think it’s a good start
The classic way of checking predictions, comparing each the observed data point to the distribution of the predicted value for that particular data point, seems better than looking at the full distribution of data and the aggregate distribution of conditional predictions (at least in the example they give of a standard linear regression). It’s also true that in some contexts “worse is better”.
Another question is the use of the “posterior” label to refer to things unrelated to the posterior distribution of parameters conditional on data in a Bayesian model. In fact the book is happy to use the word posterior without even the “conditional on data” implication. We can read in page 106 regarding “posterior predictive simulation, partial predictive simulation, and prior predictive simulation” that in reality “they are all posterior predictive simulations”.
Carlos:
There is pointwise checking, which is great, but it can miss systematic patterns.
There’s omnibus checking such as chi-squared tests, which also is great, but then there’s the challenge of figuring out the reference distribution. For normal linear models without constraints or asymptotically you can count degrees of freedom, but that won’t work in general. This problem came up in my Ph.D. thesis and was my motivation for working out posterior predictive checks: I wanted to get an appropriate reference distribution for the chi-squared test of an image-reconstruction model with positivity constraints. Plugging in the point estimate does not work in such problems. You need to correct for overfitting.
And then there’s graphical predictive checking, which I also think is great, and is what I usually do nowadays. Here, it can work to plug in a point estimate, but it depends where the point estimate comes from. For example, when fitting multilevel models, the marginal maximum likelihood estimate of the variance parameters is often on the boundary—my colleagues and I published a couple of papers on that topic—and I wouldn’t want to do a predictive check conditional on that point estimate. It could work with a posterior mode estimate using a zero-avoiding prior, though.
The larger point is that classical model checking and hypothesis testing had a big whole, which was what to do when the point estimate is noisy and you can’t do an analytic correction. I think that posterior predictive checking fills that hole nicely.
> The larger point is that classical model checking and hypothesis testing had a big whole, which was what to do when the point estimate is noisy and you can’t do an analytic correction. I think that posterior predictive checking fills that hole nicely.
Maybe. It is that’s hard to know what “posterior predictive checking” means anymore when even “prior predictive checking” is really “posterior predictive checking”. If the point estimate is noisy it seems that posterior predictive checking using the point estimate won’t be able to fill any hole.
> There is pointwise checking, which is great, but it can miss systematic patterns.
Looking at the first example given by those people using posterior predictive checks for non-Bayesian models that you mentioned the other day it’s not obvious that looking at the “Posterior Predictive Check” chart helps in finding systematic patterns that would be missed on the other charts: https://easystats.github.io/performance/reference/figures/unnamed-chunk-14-1.png The residuals charts may be more difficult to understand but also contain more information.
Carlos:
For more on posterior and prior predictive checking, I recommend you take a look at our new book, Bayesian Workflow.
Speaking more generally, I worked on Bayesian predictive checking because it allowed me to attack problems that I could not easily address using available techniques. I would not claim that any method—including my own—is all-encompassing or always better than other approaches.
I’m reading it!
Your new book Bayesian Workflow is “the book” I was referring to when I wrote the following earlier today that
> the book is happy to use the word posterior without even the “conditional on data” implication. We can read in page 106 regarding “posterior predictive simulation, partial predictive simulation, and prior predictive simulation” that in reality “they are all posterior predictive simulations”.
Thank you so much for all of your work and for making so much freely available.
Please can I ask, I am working through Regression and Other Stories, would you expect after that someone would be able to read this book or would I need to learn more explicitly Bayesian statistics before reading this?
I’m sorry if that is a silly question or you have answered that already.
I am a healthcare scientist trying to self teach these things.
Thank you so much, and thank you again for all your work.
As someone who bought the book with my own money from the publisher’s website, making the pdf freely available makes me… very happy that I chose to buy the physical book. :)
Thanks and respect for making quality learning resources freely available!
First of all, I join all the others in thanking all of you for this work, and for making it freely available!
There is a small typo in the free pdf version, at page 492: “Partial pooling for estimating the rats of a rare cancer:”. “Rats” should be “rates”, I guess.
Oh, good catch! Thanks.
“a person’s body weight is related to their volume, which is related to height. If a person were a perfect cylinder, then the relationship would be:
w = k pi r^2 h = k pi p^2 h^3.
[…] Suppose we divide the height values by the mean height in the population. This makes them dimensionless, and the mean height is now 1. We can do the same to the weight values. Since a person of average height is expected to also have an average weight, we also expect:
1 = k pi p^2 1^3,
which implies that whatever the values of k and p,the product k p^2 = 1. So instead of estimating k and p, we can just predict that the expected weight, after transformation, is the cube of the height, after transformation. No parameters are required, because the parameters in this example serve to relate the measurement scales to one another. The geometry is universal.”
A person of average height is expected to also have an average weight? That would be true, for these cylindrical people, if E[h^3] = E[h]^3 but that’s not quite right. It may be approximately right but the result doesn’t seem so “universal” after all.
Carlos:
I don’t know—I didn’t write that paragraph! But I think the general point is that this sort of modeling can be done, and it’s a step forward from the usual form of regression modeling in which predictors are just thrown in without regard to scaling.
Its true that you may not find that the avg height person has avg weight but you can choose the weight and height scales independently and then define the weight scale as the avg weight of a person of avg height.
You can also use weights and the cube of heights, or heights and the cubic root of weights, and everything is nice and linear. I don’t say the suggestion was bad, just that the justification was not good. The image of shiny happy cylindrical people was immediately replaced by that of Jensen when I read that.