Richard Artner writes:
I have two questions. The first one is, of course, related to Bayesian post-selection inference (which I first asked you about five years ago. Back then, you admitted that Bayesians are not immune to overfitting when using the same data to modify/extend a model and to predict with it. Recently, you even mentioned this issue in your new article on Bayesian workflow. However, I have not yet seen research from you (or others) that investigates the effect of a data- (and thought-driven) workflow that replaces models with better ones (better likelihoods and better priors) on overfit and poor out-of-sample predictions. After all, invalid p-values, confidence/credible intervals and biased point estimates are just the other side of the same coin (with overfit and poor predictions representing the other side of the coin). I am currently considering starting a research line that focuses on post-selection inference in Bayesian workflows and any guidance (literature ?) would be greatly appreciated.
The other question concerns the prior predictive distribution whose meaning I struggle with a lot. Henceforth, I provide a quick summary of my issues/thoughts.
A classical, parametric statistical model for measurements is a family of probability distributions P that depend on a finite number of unknown parameters (e.g., the Gaussian family for a single measurement which is determined by a location and a scale parameter). In the M-closed case, the probability distribution that corresponds to those parameter values is the true data-generating process (TDGP). A Bayesian model consists of the likelihood in a conjunction with a prior distribution. Because of this prior distribution, we can make predictions about the measurements via the prior predictive distribution, but when does it make sense to do so? The prior predictive is the marginal distribution of the measurements (i.e., the result of integrating out all model parameters in the joint distribution). Given its definition, the meaning of the prior predictive depends on the choice of prior distribution for the parameter vector. Here are three distinct interpretations of the prior distribution which result in different interpretations of the prior predictive distribution:
View 1: The prior distribution is chosen such that the prior predictive distribution corresponds to (our best guess of) the TDGP.
View 2 (subjective Bayesianism): The prior describes our beliefs about the relative plausibilities of the probability distributions in the model family P.
View 3 (your preferred view?): The prior works as a regularization device that helps mitigating the impact of sampling error on model fit. Similar to View 1, the prior serves a pragmatic goal under this view and the prior predictive can be very different from the TDGP.Under View 1, the Bayesian model is akin to a fitted classical model. This view requires very strong priors as well as hierarchies (see toy-example below). Furthermore, it can be sensible to compare Bayesian models via Bayes factors under View 1. Under Views 2 and 3, Bayes-factor tests don’t make any sense as they solely depend on the prior predictive distribution of two models (evaluated at the observed data, respectively).
Personally, I have drawn the following conclusions:
The role of the prior predictive (and the posterior predictive) distribution is unclear under Views 2 and 3.
The prior predictive can only be close to the TDGP if there is a hierarchy (e.g., a random effects model where the random effects (the parameters) are first drawn from a certain probability distribution. Afterwards, data are generated using the drawn random effect. Rinse and repeat.) We can illustrate this with a toy example: 3 six-sided dice of equal size are in a bag and one of them is drawn to generate data. One die is fair, one has a 1 on two of its sides and the third one has a 1 on three of its sides.
Situation 1: We draw a die from the bag, roll it, and report whether it showed a 1. Afterwards, we return it to the bag and repeat the process.
Situation 2: We draw one die from the bag and repeatedly roll it (and always report if it showed a 1).
For both situations, an obvious choice for a Bayesian model is a binomial likelihood with a parameter prior of P(theta=1/6)=P(theta=1/3)=P(theta=1/2)=1/3 since we know that each die has the same chance of being drawn. In Situation 1, the prior predictive is the TDGP. In situation 2, it is not. In situation 2, the prior predictive is very different from the TDGP regardless of the die drawn. Nevertheless, our Bayesian model is arguably the best possible model for this toy example (at least from View 2).Given the above observations/conclusions, I find it difficult to agree with conclusions such as Bayes factors measure prior predictive performance and Bayes factors evaluate priors, cross validations evaluate posteriors. To me, the conclusions drawn by Bob Carpenter in those two blog posts only hold under View 1 (which is really only possible in special cases that involve hierarchies) but not under Views 2 and 3.
How do you interpret the prior predictive? What is its relation to the TDGP? When are Views 1, 2, and 3 reasonable? Are there other relevant views on the prior predictive that are not considered here?
Given that I am unsure about the meaning of the prior predictive, I also struggle with the meaning of the posterior predictive. In particular, it is unclear to me why one would want to make predictions via the posterior predictive under views 2 and 3. If there is little available data, it will be similar to the prior predictive and very different to the TDGP. If there is plenty of data, we could use the posterior predictive and we would be fine (due to the Bernstein-van Mises theorem), but we could also just work with posterior quantiles in that case. Isn’t it always better to predict using posterior quantiles (e.g., the median for our best guess and the .1 and .9 quantile to express uncertainty) instead of the posterior predictive under View 3?
My reply:
In answer to the first question about workflow and post-selection inference, I’d recommend that you look at model-building workflow (building models, checking models, expanding models) as part of a larger scientific workflow, which also includes substantive theory, measurement, and data collection. In statistics and machine learning—theory and application alike—we focus so much on the analysis of some particular dataset in the context of some existing theory. But real science and engineering almost always involves designing new experiments, incorporating new data into our analyses, and trying out new substantive models. From that perspective, looking at post-selection inference is fine—it represents a sort of minimal adjustment of an analysis, in the same way that the sampling standard error from a survey is a minimal statement of uncertainty, representing uncertainty under ideal conditions, not real conditions.
In my own applied work, the sorts of adjustments that would be needed to adjust for model selection would be pretty minor, so it’s not where I focus my theoretical or methodological research efforts.
For bad analyses, it’s another story. For example, if Daryl Bem of those notorious ESP papers were to have used Bayesian methods, he still could’ve selected the hell out of things and found big fat posterior probabilities. But then I think the appropriate way to address this problem would not be to take these (hypothetical) apparently strong results and adjust them for model selection. Rather, I would recommend modeling all the data together using some sort of hierarchical model, which would have the effect of partially pooling estimates toward zero.
To me, doing poorly-motivated model selection and then trying to clean it up statistically is kinda like making a big mess and then trying to clean it up, or blowing something up and then trying to put it back together. I’d rather try to do something reasonable in the first place. And then, yes, there are still selection issues—there’s not one single reasonable hierarchical model, or only one single reasonable regularized machine learning algorithm, or whatever—but the selection becomes a much smaller part of the problem, which in practice gets subsumed by multiple starting points, cross validation, new data and theories, external validation, etc.
In answer to the second question about prior and predictive distributions, let me start by correcting this statement of yours: “A Bayesian model consists of the likelihood in a conjunction with a prior distribution.” The more accurate way to put this is: A Bayesian model consists of a data model in a conjunction with a prior distribution. The data model is the family of probability distributions p(y|theta). The likelihood is p(y|theta), considered as a function of theta for the observed data, y. As discussed in chapters 6, 7, and 8 of BDA3, the data model and the likelihood are not the same thing. There can be many data models that correspond to the same likelihood function. For Bayesian inference conditional on the data and model, you don’t need the data model + prior, you only need the likelihood + prior. But for model checking—prior predictive checking, posterior predictive checking, and everything in between (really, all of this can be considered as different forms of posterior predictive checking, conditioning on different things)—the likelihood isn’t enough; you need the data model too. Again, we have some examples of this in BDA, the simplest of which is a switch from a binomial sampling model to a negative-binomial sampling model with the same likelihood but different predictive distributions.
You ask, “when does it make sense” to make prior or posterior predictions. My answer is that you can interpret such predictions directly, in a literal sense as predictions of hypothetical future or alternative data based on the model. Suppose you have a hierarchical model with modeled data y, local parameters alpha, hyperparameters phi, and unmodeled data x, and your posterior distribution is p(alpha,phi|x,y) proportional to p(phi|x)p(alpha|phi,x)p(y|alpha,phi,x). Just to fix ideas, think of alpha as corresponding to an “urn” from which the data y are drawn, and think of phi as a “room” that is full of urns, each of which corresponds to a different value of alpha. Finally, think of the prior distribution of phi as a “building” full of rooms. The building is your model. – So the generative model is: Go to the building, sample a room at random from that building, then sample an urn at random from the urns in that room, then sample data y from your urn.
Suppose you fit the model and obtain posterior simulations (alpha,phi)^s, s=1,…,S.
– You can simulate new data y* from the prior predictive distribution, which would correspond to picking a new room, a new urn, and new data. For each simulation s, you can do this by drawing phi* from p(phi|x), then drawing alpha* from p(alpha|phi*,x), then drawing y* from p(y|alpha*,phi*,x).
– Or you can simulate new data y* from the posterior predictive distribution for new data from new groups, which would correspond to staying in the same room but then drawing a new urn and new data from that urn. For each simulation s, you can do this by keeping phi^s, then drawing alpha* from p(alpha|phi^s,x), then drawing y* from p(y|alpha*,phi^s,x).
– Or you can simulate new data y* from the posterior predictive distribution for new data from existing groups, which would correspond to staying in the same room and keeping the same urn and then drawing new data from that urn. For each simulation s, you can do this by keeping phi^s, keeping alpha^s, then drawing y* from p(y|alpha^s,phi^s,x).
These three different distributions correspond to different scenarios. They can all be interpreted directly. This has nothing to do with “relative plausibility” or whatever; they’re just predictive distributions of what you might see in alternative rooms and urns, if the model were true.
It’s not necessary to describe this all using multilevel models—you can distinguish between prior and posterior predictive checks with a simple model with just y and theta—but I find the multilevel modeling framework to be helpful in that it allows me to better visualize the different sorts of replications being considered. You could also consider other distributions in a non-nested model, for example predictive data on new patients, new time periods, new hospitals, etc.
Regarding Bob’s posts about Bayes factors and prior predictive checks: I take his point to be not philosophical but mathematical. His point is that the Bayes factor is mathematically an integration over the prior predictive distribution. This is obvious—you can just look at the integral—but it seems that people get confused about the Bayes factor because they look at it in terms of what is supposed to do (give the posterior probability that a model is true, something that I think is typically the wrong question to ask, for reasons discussed in chapter 7 of BDA3 and also this article with Shalizi) rather than what it does. In that sense, Bob’s post fits into a long tradition of statistical research and exposition including Neyman, Tukey, and others who work to understand methods in terms of what they actually do. This does not address your questions about prior and posterior predictive distributions—for that I refer you to the above paragraph about rooms and urns, which is largely drawn from my 1996 paper with Meng and Stern—; you just have to read Bob’s posts literally, not as Bayesian positions but as agnostic “machine learning” descriptions of what prior and posterior predictive checks do.
> View 1: The prior distribution is chosen such that the prior predictive distribution corresponds to (our best guess of) the TDGP.
> View 2 (subjective Bayesianism): The prior describes our beliefs about the relative plausibilities of the probability distributions in the model family P.
Wouldn’t “our beliefs about the relative plausibilities” be related to “our best guess” of what’s going on?
> Situation 1: We draw a die from the bag, roll it, and report whether it showed a 1. Afterwards, we return it to the bag and repeat the process.
> Situation 2: We draw one die from the bag and repeatedly roll it (and always report if it showed a 1).
> For both situations, an obvious choice for a Bayesian model is […]
Maybe it’s not surprising that using the same model in two different situations doesn’t work too well. The data is a sequence and the prior predictive distribution is over sequences. In the first case each draw is independent and we have just the product of single draw distributions. In the second case the single draw distribution is valid for any single draw but there are correlations.
Nothing prevents us from calculating the prior predictive distribution for sequences of any length. For a single event p([T])=1/3 and p([F])=2/3 in both cases. But for longer sequences the predictions are different. In the first case p([T T])=1/9, p([T F])=p([F T])=2/9 and p([F F])=4/9. However, in the second case [T T] and [F F] are more probable than in the first case (while [T F] and [F T] are less likely).
I think the confusion comes from view 1. View 1 is just flat out wrong.
The prior predictive distribution for, say, x represents what we know about x before seeing the data. It isn’t our best guess of the true data generating process, though.
Consider the case of trick coin that has two faces that are the same. If we know nothing else about the coin, our prior predictive for the outcome of a toss would be
p(H) = p(T) = 1/2
That represents all we know about the possible outcome. This is certainly not though a sensible estimate for the true data generating process, though, which is either p(T) = 1 or p(H) = 1, and cannot possibly be p(H) = p(T) = 1/2.
To consider our best guess for the true data generating process, we need to think of a distribution on possible true data generating processes – a distribution over distributions. If we had to, we could summarise that by e.g. the mode or mean. The prior predictive distribution is equal to the mean. Just like the other uses of the mean (e.g, the posterior mean), the mean might not be a good choice and needn’t be plausible or even possible.
Fwiw:
Definition 1 is not “right” or “wrong”; it’s a definition. It’s a tail-area probability, the probability under the model that you’d see something as extreme as the data, if the model were true. I agree that the tail-area doesn’t, or shouldn’t, have much practical use—but people do use it. It’s one of the things that people mean when they talk about the p-value, and it’s an accurate description of certain things that people do.
Wrong post. That comment is about 1 “View 1: The prior distribution is chosen such that the prior predictive distribution corresponds to (our best guess of) the TDGP.”
oh, my bad
You’re right. View 1 seems fundamentally incompatible with the use of probability distributions to model uncertainty about the true value of the parameters of the model (i.e. about the true data generation process which the model stands for).
The prior predictive distribution doesn’t corresponds to (our best guess of) the TDGP. It corresponds to many – possible an infinite number of – DGPs and at most one will be true.
The prior predictive distribution is a probability distribution of the data that does not depend on any unknown parameters. How could it possibly correspond to “many – possibly an infinite number of – DGPs and at most one will be true”? A data model along with some point in the parameter space corresponds to a DGP and in the M-closed case, there is one point in the parameter space that turns the data model into the TDGP.
Most importantly, what is the purpose of prior predictive checks if one does not hold View 1? Just to be clear, I do not hold View 1 which is why I do not see any value in prior predictive checks as a means to “test the model” or “look for model misfit” (which is also why I do not understand the logic of hypothesis tests via Bayes factors).
> The prior predictive distribution […] does not depend on any unknown parameters.
I may have misunderstood everything you wrote.
> Given its definition, the meaning of the prior predictive depends on the choice of prior distribution for the parameter vector.
> View 1: The prior distribution is chosen such that the prior predictive distribution corresponds to (our best guess of) the TDGP.
Did you mean that in view 1 the prior distribution is chosen to be just a single point in the parameter space?
Ok, let me clarify. There are situations were the data model does not correspond to the TDPG for any point in the parameter space but the prior predictive represents the TDGP. These cases are quite rare and involve hierarchies (an example of this is Situation 1 in my toy example).
My main point is that I don’t understand the usefulness of prior predictive checks in situations where the data model turns into the TDGP for one specific point in the parameter space. An example of this is y ~ N(theta,1) with some prior on the standardized average treatment effect (e.g., p(theta) ~ N(0.2,0.6)). If the true average effect is theta*=0.3, the TDGP is y ~ N(0.3,1) which is very different from the prior predictive distribution.
I think one should distinguish statistical inference problems from straight-foward probability problems. Your ‘View 1’ considerations seem to be applicable to the latter but not to the former.
In statistical inference problems there is a model which is not fully specified (so we have a prior probability distribution for the uncertain paramaters, and a corresponding predictive probability distribution for the yet unobserved data, and a posterior probability distribution for the uncertain parameters after some data is observed, and a corresponding predictive probability distribution for future data that takes into account the previously observed data, etc.).
In purely probabilistic problems the model is fully specified. The only uncertain thing is the data but the predictive probability distribution for the outcomes is fixed. The model won’t ever change. There are no prior/posterior distributions for the parameters of the model which are known with certainty.
The following problem (a slightly simplified but still interesting version of your problem) is of the second kind. It’s a high school level probability problem – not a statical inference problem.
“Two six-sided dice are in a bag. One has one ‘1’, the other has three ‘1’s. We draw a die from the bag, roll it and report whether it showed a 1. Afterwards, we return it to the bag and repeat the process. What is the distribution probability for the possible oucomes?”
It’s easy to calculate that for one roll there are two possible outcomes, ONE and NOTONE, which happen each with probability p(ONE) = 1/3 and p(NOTONE) = 2/3. That’s all. High school students can solve it. They will not call that a “prior” “predictive” probability but what would that mean anyway? Prior as opposed to what? Predictive as opposed to what? It’s also easy to calculate the probability for sequences of outcomes because there are independent: p(ONE/ONE) = 1/9, p(ONE/NOTONE) = P(NOTONE/ONE) = 2/9, p(NOTONE/NOTONE) = 4/9, p(ONE/ONE/ONE) = 1/27, etc.
The variant where the die is not returned to the bag but rolled again and again is only slightly more complex. It’s still a high school probability problem if the analysis is ex-ante (the calculations are conditial only on the problem statement: we will draw a die from the bag, etc.). It’s not different conceptually from the first problem and the answer differs only in the correlation between rolls. What’s the distribution probability for the possible outcomes for the first roll? p(ONE) = 1/3 and p(NOTONE) = 2/3. What’s the distribution probability for the possible outcomes for the second roll? p(ONE) = 1/3 and p(NOTONE) = 2/3. One can also calculate the probability for sequences of outcomes, even though it’s not just a matter of multipliying single-outcome probabilities. p(ONE/ONE) = 5/36, p(ONE/NOTONE) = p(NOTONE/ONE) = 7/36, p(NOTONE/NOTONE) = 17/36, p(ONE/ONE/ONE) = 14/216, etc. The probability of getting two ones in two rolls is higher than before (5/36 > 4/36) and also the probability of not getting any (17/36 > 16/36).
Of course, the second problem may also be treated as an inference problem. The model changes and no longer consists of a bag with two dies. It’s a single die now, with an unknown number of ‘1’s on it. We have a prior distribution for the parameter p(one ‘1’)=p(three ‘1’s)=1/2. (It’s not just a subjective guess – here it’s the actual frequency of the parameter according to the problem statement.) Now we can update the distribution for the parameters whenever we observe an outcome and calculate the probability distribution for unobserved outcomes conditional on that information. I could for example roll it once without looking at it, roll it again, and calculate a probability distribution for the outcome of the first roll (which I ignore) taking into account the outcome of the second roll (that I know).
In the first problem, however, it makes no sense to consider an inference problem to predict what happens if we roll the die again because we say we don’t – and if we do it becomes the second problem.
Thanks for your answers, Andrew! I have two follow-up comments/questions regarding the prior predictive.
1) I see what you mean by mathematical as opposed to philosophical and I actually agree 100% with the math that Bob provided (how could anyone not?). It is the philosophical interpretation that I am concerned with. Specifically, the distinction between the (prior predictive) performance (i.e., the predictive performance of the prior predictive distribution) and the prior, predictive performance (which is the performance of the predictions made by the Bayesian model prior to fitting it to (new) data). The latter depends on how one uses a Bayesian model to predict data. Using the prior predictive distribution is one way of doing so, but there are others. Personally, I believe that our best guess of the TDGP corresponds to the best way of predicting (but I would love to read counterarguments for this claim). As a consequence, using the prior predictive distribution for predictions aligns with View 1. More generally, prior/posterior predictive checks align with View 1. After all, why would simulated data from the prior/posterior predictive look similar to actual data unless they actually do represent the TDGP?
2) Your example is a more complex version of my Situation 1 (with urns replacing dice) that involves not just one but two layers of hierarchy. I was able to follow your reasoning. However, it is Situation 2 (no hierarchies) where I fail to place meaning on the prior predictive distribution. As a realistic example consider the average treatment effect of a drug in a fixed population: A normal data model y ~ N(theta,sigma²) with some prior on the standardized average treatment effect (e.g., p(theta) ~ N(0.2,0.6)). Irrespective of the true value of theta, the prior predictive will provide poor predictions for y in this scenario. However, I am unwilling to label our Bayesian model as ‘bad’, which is how I interpret your statement: ‘I like to look at the prior predictive distribution to remind myself that my model does not make any sense’.
If your model is narratively generative, in the sense that you’re building a model of what you believe is the data generating process (as opposed to some sort of heuristic regression, for example), then it seems to me that your “view 1” is a natural consequence of “view 2”. That the prior predictive may have poor predictions of y would indicate that either the generative model is inadequate or the prior is poorly chosen (either of which should call into question one’s assumptions regarding domain expertise (ie subjective beliefs, although people prefer ‘knowledge’ or ‘expertise’) for both the generative model and/or the prior). But that’s what the observational model is for. The goal of prior predictive checks is not to compare to the observed data, but rather predict data from the model that doesn’t violate one’s domain expertise (knowledge, beliefs). If you are comparing your prior predictive checks to the observed data for the purpose of informing/updating priors, then you are double dipping. In complex models, the prior of each individual parameter is not so easy to think about in terms of its ramifications to the model as a whole. It is much easier to think in terms of the outcome space – something that we are familiar with. However, this outcome space should not be defined by our observations, as you seem to suggest. Rather it should be defined by the same knowledge/expertise/beliefs that defined the generative models and priors in the first place. It’s an internal check. Not an external one.
This is a really long post where I don’t even understand what the question really is. But maybe can offer some help in interpreting prior and posterior predictives.
A prior predictive distribution tells us what our prior parameters imply about which data points would be in the high probability region of our model.
If I have a model say of rainfall in some desert somewhere and there are parameters describing it which I am uncertain about, then using the prior predictive I can find out what levels of rainfall are within the realm of plausible vs which are outside that realm.
A prior predictive can be extremely useful to check whether your priors as expressed do a good job of implying outcomes you think are reasonable. If the high probability region of the desert rainfall includes amounts of rain that exceed most rainforests you can be pretty sure your priors are wrong.
In fact, in many models I express some basic range priors on parameters and then add a factor to the model which functions as a dependency factor that acts in terms of the predictive quantities.
In Turing
a ~ Normal(0,1)
b ~ Normal(0,1)
addlogprob!(logpdf(sqrt(a^2+b^2),Normal(1,0.2))
It says that a,b are in the general region near 0 but also the radius from 0 to (a,b) is quite close to 1.
The radius here is maybe like a predictive distribution. Maybe radius is what we measure in our data. Maybe I know it’s near 1 because of background knowledge. So let’s include that knowledge in the model!
So, prior predictive distributions are meaningful for helping us understand our priors.
Then, you ask about posterior predictive shouldn’t we just use “a median” … I invite you to tell me what is “the median of (a,b)”
Posterior predictive distributions in general aren’t about a single dimensional outcome. I might be interested in something like the map of rainfall over an entire peninsula. Posterior predictive maps will be functions of x,y.
Hope that helps!
speaking of post-selection inference, has anyone used the selectiveInference package in R ? I don’t understand what it is doing. It gives coefficient point estimates that are in between the shrunken lasso coefficients and the completely unshrunken classical glm. Is this expected behavior ?