Prob Prog 2021 just ended. Prob Prog is the big (250 registered attendees, and as many as 180 actually online at one point) probabilistic programming conference. It’s a very broadly scoped conference.
The online version this year went very smoothly. It ran a different schedule every day to accommodate different time zones. So I wound up missing the Thursday talks other than the posters because of the early start. There was a nice amount of space between sessions to hang out in the break rooms and chat.
Given that there’s no publication for this conference, I thought I’d share my slides here. The talks should go up on YouTube at some point.
Slides: What do we need from a PPL to support Bayesian workflow?
There was a lot of nice discussion around bits of workflow we don’t really discuss in the paper or book: how to manage file names for multiple models, how to share work among distributed teammates, how to put models into production and keep them updated for new data. In my talk, I brought up issues others have to deal with like privacy or intellectual property concerns.
My main focus was on modularity. After talking to a bunch of people after my talk, I still don’t think we have any reasonable methodology as a field to test out components of a probabilistic program that are between the level of a density we can unit test and a full model we can subject to our whole battery of workflow tests. How would we go about just testing a custom GP prior or spatio-temporal model component? There’s not even a way to represent such a module in Stan, which was the motivation for Maria Gorinova‘s work on SlicStan. Ryan Bernstein (a Stan developer and Gelman student) is also working on IDE-like tools that provide a new language for expressing a range of models.
Then Eli Bingham (of Pyro fame) dropped the big question: is there any hope we could use something like these PPLs to develop a scalable, global climate model? Turns out that we don’t even know how they vet the incredibly complicated components of these models. Just the soil carbon models are more complicated than most of the PK/PD models we fit and they’re one of the simplest parts of these models.
I submitted two abstracts this year and then they invited me to do a plenary session and I decided to focus on the first.
Paper submission 1: What do we need from a probabilistic programming language to support Bayesian workflow?
Paper submission 2: Lambdas, tuples, ragged arrays, and complex numbers in Stan
P.S. Andrew: have you considered just choosing another theme at random? It’s hard to imagine it’d be harder to read than this one.
1. As a linguist, you’ll perhaps enjoy this line from David Owen: “of course at any convention there is only one attendee, the convention itself, while the people in attendance are attenders,” from his entertaining report from the annual convention of Meeting Planners International—yes, that’s a meeting of meeting planners.
2. In answer to the question in your P.S.: You will perhaps not be surprised to know that I have no idea how to change the theme. The changes that have been made have been by our sysadmin. Our current plan is to pay a web designer at Columbia to make something that we’re all happy with. But . . . you have an Administrator account on this blog, just like me. So if you want to change the theme, go for it! My only request is that you keep the following plug-ins: anti-spam and recaptcha (our sysadmin tells me these are necessary to avoid us being overwhelmed), auto-tweeting, and classic editor (which is what I use to write my posts). Go for it!
Hi Bob, I really enjoyed your talk, and thought it was spot on.
I’ve been thinking that it would be useful to have a language whose sole purpose is to precisely describe models (joint probability distributions) at a fairly high level of abstraction, with well-defined semantics, with clarity an important language design priority, and without regards to any particular implementation strategy, nor any need to specify up front what variables are observations. The tools for this language would do things like type checking and other sorts of syntactic correctness checking, and maybe output an abstract syntax tree, but there would be no attempt to implement inference. The purposes would be:
1) Communication, including specification.
2) To learn more about how to express and manage models in a modular fashion that encourages reuse of model components, expressiveness and clarity, etc.
3) To allow one to build up a collection of PPL-agnostic models to serve both as cases for comparing PPLs and as starting points for anyone wanting to implement the models in their favorite PPL.
4) To serve as a target for experiments with automating various kinds of model transformations (such as reparameterization).
5) To be used as input to source-to-source translators that might produce Stan models, or Pyro models, etc., or Monte Carlo simulators, etc. Of course, a translator might return with “Can’t implement this model in this language,” or it might need some hints, separate from the source model, on how to implement it efficiently. And you’d some sort of conditioning operator to indicate what the observed data are in a particular application.
I don’t think this is what you mean, but I’ve been making use of the new stan generated quantities command combined with #include blocks to dispatch arbitrary parameters to stan models in testing. This workflow is currently a little awkward since cmdstan requires a bunch of stan diagnostic comments in the parameters csv or it spits out an error.
Thanks for reporting, I enjoyed reading it.
One of the things I’ve been thinking a lot about recently in the context of workflow and would be happy to get your feedback on is the implementation of prior simulations. I am currently of the opinion that auto-generating prior simulator from model code is of limited value for model _development_. The main limitation IMHO is that if your model computes a different density than what you think it computes (e.g. because of a bug in the code), the auto-generated prior simulation will match the incorrect density and it will appear that you can recover the simulated parameters and more generally, all SBC-like checks will pass.
My current personal workflow thus involves having a separate implementation of the simulator and I’ve found it beneficial to have the simulator implement the data-generating process in a different way than the model does (e.g. model in Stan vs. simulator in R; matrix multiplication in model vs. loops in simulator; a parameter marginalized out in the model vs. explicitly including the parameter in simulator). This can be traced to the tradition of double-implementing critical elements of software systems and has so far proven quite effective at catching bugs/my misunderstandings with relatively little added effort. Also in many cases it is not clear that a simulator could even in principle be auto-generated for more complex models with e.g. marginalization/nuisance parameters. Practical examples on how catching bugs might look like are worked-out in the vignettes for the SBC package (https://hyunjimoon.github.io/SBC/articles/index.html) if anyone’s interested.
There are definitely issues that an auto-generated prior simulation will catch (e.g. divergences). Also if one is testing algorithms, then being able to automatically extract simulators for SBC is definitely useful and the situation is diffferent – so it is definitely worthwhile to aim for having it at some point. I just currently don’t have a lot of use for it in my workflow.
Would be glad to get your comments on that! Thanks!