Yee Whye Teh sends along this paper with Leonard Hasenclever, Thibaut Lienart, Sebastian Vollmer, Stefan Webb, Balaji Lakshminarayanan, and Charles Blundell. I haven’t read it in detail but they not similarities to our “expectation propagation as a way of life” paper. But their work is much more advanced than ours.
Convergence guarantee! This looks very exciting… I just need someone to explain the details to me :)
The “posterior server” should be renamed “Big Posterior” to match “Big Data”
I had a bit of trouble reading this paper. It invokes quite a few duality arguments and exponential family tricks from both Wainwright and Jordan (2008) and Minka (2004). My understanding of the algorithm is that they 1. consider a general class of probability models in which you can compute individual log-likelihood terms and the prior density; 2. consider a fully factorized approximating family whose univariate distributions form an exponential family; and 3. solve the double loop variational objective by approximating the tilted distribution with MCMC and optimizing local KL(p||q)’s with natural gradient steps on three sets of parameters: the mean parameters of the exponential family, the auxiliary parameters that solve for the convex conjugate of the exponential family’s log-normalizer, and the Lagrange multipliers that convert the problem from constrained to unconstrained.
The double loop lets them appeal to other work on convergence guarantees. The exponential family lets them leverage natural gradients. I think it still has the EP as a way of life’s formulation as carving out a data set, enabling separate analyses, and then aggregating them. The double loop stuff clouds a lot of these connections as I’m trying to understand it though.
Dustin:
Whenever I see the word “guarantee” in a statistics paper I just turn off my brain for a moment until the word goes away from my vision.
Yes the term is overloaded and deceptive in this literature. So is “provable”. :)
I love this comment.
In grad school I read a bunch of stuff about asymptotic series. There’s all these great ways to accurately calculate the values of various functions. They all converge with increasing number of terms, UP TO SOME POINT and then diverge. Most of them arrive at very high accuracy at only a few terms, like maybe 2 or 3, whereas convergent series for the same result might take 10 or 100 or 1000 terms. All those 1000 term series have convergence guarantees. The result you want to use is the one that’s accurate at 3 terms and has a DIVERGENCE guarantee. :-)
Pardon my French, but isn’t “they not similarities” a typo for “they note similarities”…?