The Computational Abstractions for Probabilistic and Differentiable Programming (CAPP) Workshop is happening this Friday, 29 July 2022. It’s organized out of Cambridge and is running on UK time.
Program and Zoom link
The schedule of speakers is online and the free Zoom link is available under the Location tab of their web site:
Despite the title, the talks sound very practical. The speaker lineup includes developers from BUGS, JAGS, Stan, JAX, and Turing.jl.
My talk
I (Bob) will be talking about our plan to introduce Python-like comprehensions into the Stan language to deal with constant matrix constructions like Gaussian Process covariance matrices. This is motivated by our shift in underlying matrix data structure for autodiff in Stan—what the Stan developers have been calling “mat-var” in meetings and pull requests because it’s an autodiff variable with a matrix value and matrix adjoint rather than our original approach using a matrix of scalar autodiff variables.
P.S. This is a big reduction in memory pressure and time for larger-scale matrix ops for Stan. The really clever part is how the C++ devs (Tadej Ciglarič, Steve Bronder, Rok Češnovar, et al.) have managed to abstract the mat-var construct in a way that’s backward compatible with our existing autodiff with simpler code. If you like C++ and functional template meta programming in C++, I urge you to follow along. Steve’s about to drop all the dev-facing doc for all of this—it’s currently in a massive PR undergoing final revisions.
Bob:
This sounds really cool—thanks for sharing. And it reminds me that I want to write that informal spec showing what I’m envisioning, regarding a more compact syntax for some Stan models.
“our plan to introduce Python-like comprehensions into the Stan language”
Great idea, but I feel compelled to point out that list comprehensions are a feature that Python borrowed (in 2000) from Haskell, which had them since the Haskell 1.0 Report in 1990. Haskell in turn borrowed them from Miranda, which was created around 1985. Seems odd to me to associate them with latecomer Python.
Wow — Miranda is still around. The latest release was in 2020. It was a historically important language, but I had no idea it was still being maintained. https://www.cs.kent.ac.uk/people/staff/dat/miranda/.
I learned about comprehensions from set theory, where they were widely used by the late 1800s. Unfettered set comprehension led to Russell’s paradox, which was remedied with the well-founded comprehension of Zermelo-Fraenkel (ZF) set theory in the early 1900s.
I somehow didn’t think either Haskell or ZF set theory were going to be on data scientists’ radar, so I stuck to the more familiar.
For those who are interested, there’s more history in the Wikipedia article on list comprehensions. I was amused to see that POP2 (a functional PL from the 1970s) called them “ZF expressions.”
The memory debug functional calls it now Struct-of-Arrays (SoA) which I was told to be more commonly used.
The new data structure provides 25%-40% reduction of sampling time for several posteriors that I tested. The models included GLMs with many predictors, GPs with basis functions, and GPs with covariance matrices. These all had vector or array parameters with more than tens of elements.
I’m excited to see further speedups from constant matrix constructions for GPs and GLMMs.
Is the impact primarily GPs and GLMMs? Would there be any way to speed up models with something like a multivariate normal distribution with time-varying correlations?
Probably. If you post a Stan code example to Stan discourse https://discourse.mc-stan.org/ and ping me, I can have a look
The way the previous post title and the next post title are juxtaposed at the top of this blog entry make the job sound a lot more interesting than it probably is.