Here’s what Anirban Bhattacharya, Antonio Linero, and Chris Oates have to say:
Grand Challenge 1: Understanding the Role of Parametrisation
Grand Challenge 2: Community Benchmarks
Grand Challenge 3: Reliable Assessment of Posterior Approximations
They also mention “software support for the full Bayesian workflow,” which I agree is super-important.
I’d also add
– Scalable computing: being able to fit the models you want to fit, on large datasets
– Scalable modeling: being able to build bigger models as you get more data. We can easily expand models in some directions—adding predictors in a regression, adding terms to a spline model, adding layers to a neural net, etc.—but, in other ways, model expansion can be a challenge. Even for something as simple as multilevel regression and poststratification, we can quickly get tangled in how to program up interactions.
There are many more grand challenges in Bayesian computation. Feel free to put your ideas in the comments.
P.S. Vaguely relevant here is the article by Aki and me, “What are the most important statistical ideas of the past 50 years?” Kind of amazing that we wrote that article five years ago. Time flies!
I think a major challenge is modeling extremely complex dependent data.
Two examples
1)Text. For a long time the best models were Bayesian or nearly so
(word trigrams maybe with some grammatical overlay) but are now dominated
by LLM deep learning which is not really Bayesian. Can Bayesians catch up?
2) Genomics (my field)
Genomes have an extremely complex dependence structure
and nobody that I know of even tries to use the full structure
of what is known. Bayesian analysis is important (for example ABC
modeling) but only partial. The priors are unrealistic in ways that
can be important.
Nick:
I think of llms and deep nets as Bayesian modeling, where the computation is done approximately.
LLMs are kind of Bayesian but I think you would be hard pressed to
write down an explicit prior. The situation in genetics is subtle.
There are quite reasonable hierarchical priors we can fairly readily
sample. but computing a likelihood or accurate sampling from the
posterior is intractable.
Nick:
I don’t know enough about llms or genetics to say anything about the priors. Speaking in general terms, deep nets are fit with lots of regularization, and I assume the regularization can be considered as approximate priors, along with inference for hyperparameters which is done in some way that we could consider as approximately Bayesian. I had the impression that these models can be run “forward,” i.e., generatively, and if there’s a generative joint model, then, in principle, one can do Bayesian inference. That said, beyond the practical challenges, there’s the complication that many users and many developers of methods don’t feel comfortable with Bayesian inference, so the inference that is done is not explicitly an approximation to a Bayesian posterior distribution. In some settings I think it can be helpful to frame a non-Bayesian procedure as approximately Bayesian–this can give insight about the implicit model (the prior and the data model) and also can lead to improvements in the inference algorithm. In some setting, everything might be so complicated that a fully Bayesian framing might be more trouble than it’s worth.
Link to “What are the most important statistical ideas of the past 50 years?” is broken!
Link fixed; thanks.
As an applied economist mostly working with R in the world of microeconometrics/quantitative market analysis:
– I’m not sure how well developed the computation infrastructure is for Bayesian microeconometrics and also for simultaneous equations models/more complex multi-equation social/behavioural sciences models.
Maybe I’m missing something, but my sense is that most of what’s available is for single equation linear models or glm’s.
Yet, causal economic analysis in business calls out for more solid Bayesian inference to handle parameter uncertainty (as opposed to cheap but inaccurate shortcuts such as Bayesian interpretations of asymptotic classic MLE confidence intervals, ad-hoc model combinations and use of regularised regression+bootstrap to get some sort of pseudo-bayesian inference). Not sure we have that now?
There’s an interesting contrast with macroeconometrics, where for all the flaws of DSGE models, their bayesian estimation and analysis was essentially standardised with packages such as Dynare that readily perform full Bayesian inference on medium-scale multi-equations macroeconomic models. An equivalent tool for say joint models of market demand and pricing/supply would be nice to have.
Daniel:
These simultaneous models can be fit in Stan, but, yes, they’ve been fit one at a time by different people. It would be useful to have a package collecting a bunch of them in one place.