Stan saves Australians $20 billion

Jim Savage writes:

Not sure if you knew, but Stan was used in the Australian Productivity Commission’s review of the Australian retirement savings system. Their review will likely affect the regulation on $2 trillion of retirement savings, possibly saving Australians around $20-50 billion in fees over the next decade.

OK, we can now officially say that Stan, as an open-source software, has recouped its societal investment.

Tomorrow’s post: What’s the evidence on the effectiveness of psychotherapy?

36 thoughts on “Stan saves Australians $20 billion

    • Hey Matt! In our original papers that generated a lot of the hubbub that led to the PC’s review, we used R and Excel only. This is the better of the two papers: https://grattan.edu.au/wp-content/uploads/2015/04/821-super-savings2.pdf

      I agree that most of the main insights from that paper don’t really require any sophisticated modeling, though we did use quite a bit of sophisticated modeling to work out what simple plots we needed to create. The PC’s review linked in the post used Stan to ask a subtly different question though: we tend to see lower marginal operating costs for larger funds. But as funds become larger, do they realize these implicit economies of scale? And how big is the heterogeneity across funds and fund types that results in these economies of scale? This is a question that really needs a hierarchical model to answer well; answering this with the raw data would suffer from issues with non-random dropout of funds from the panel.

      I should give a shout-out to Hans Zhu, the analyst behind a lot of the hard modeling work in this report. He’s now an econ PhD student at Northwestern– keep your eyes peeled, he’s very sharp.

    • So, this comment is a very good point, and it made me actually download the report and skim it.

      The main issue in the report seems to be measurement error and imputation issues. The need to account for things like people reporting “zero cost” when obviously that isn’t right. It might be right in some accounting sense (the sense in which the fund itself didn’t classify costs into a certain category) but it’s not right in an economic sense.

      Another issue is survivorship bias and the affect of consolidation.

      I think both of those things make typical simple linear regressions or even maximum likelihood type fitting unrealistic. I do think that Bayesian models here are likely to provide dramatic benefits in terms of the type of model you can fit, and its ability to impute missing data in a reasonable way with reasonable estimates of variation of those imputations, leading to better decision making…

      but obviously, even if you acknowledge the need to do the Bayesian model, you could have fit it using say Geyer’s “mcmc” package, or maybe using JAGS if the model is simple enough… and you could have just run that for long periods on multiple parallel chains, and likely have gotten a decent fit.

      Still, I think Stan’s contribution is probably at least worth a basis point (1/100 of 1% or .0001 fraction) of the cost savings. Most likely a major reason they even tried to build this model was the availability of Stan to make it relatively easy to do the model development, debugging, and fitting.

      • ad-hoc calculation to justify present value:

        Suppose you’re going to receive some policy benefit at some future point in time t, and Stan comes along and allows you to begin arguing for that policy benefit say 6 months earlier or 0.5 years… suppose further than the relevant investment discount rate is a moderate 4%

        The net present value of the savings is increased then by exp(-.04*(-.5)) ~ 1.0202 so about 2% increase in savings due to the acceleration of the research required by 0.5 yrs… So what acceleration should Stan have provided to be worth at least a basis point:

        exp(.04*dt) = 1.0001, dt = .0025 years, or about 1 day… I promise you Stan saved them *far more* than 1 day.

    • Came here to say this.

      Review saved $20-$50 billion.
      Stan was involved in the Review.
      Therefore, Stan saved $20-$50 billion.

      AWOOOOOOOGAH!!!

      The economic Klaxon is deafening.

      Nope, nope, nope, nope.

      • It wasn’t just “used in” though, most likely it was really essential to have the Bayesian model, and practically speaking it was very necessary… But you’re right… Stan should get $20M and Jim Savage and colleagues should get say half of the $20B ;-)

        • Almost $3T, with average fees above 1%–40-50bps in admin, and the rest investment. The TSP, available to US federal workers, has all-in fees of around 0.015%. A TSP in Australia probably wouldn’t work too well–see our report–so you wouldn’t want that. But you wouldn’t need to do much to remove 1/3-1/2 of the total costs in superannuation. Over 10 years, that’s a lotta dough.

        • At first, I was suspicious of the $20-$50 billion (over ten years) number too.

          But then I saw that they have $2 trillion under management. Holy moly that’s a lot of money. Maybe shaving a basis point or two off fees isn’t ridiculous.

  1. Wow! Quite apart from the fact that STAN was used, a look at the document shows how stats and science, if used educationally and with sufficient detail and explanation, can make policy evaluation so much better and more transparent! Wish that ALL policy implementations were evaluated in such a rigorous, yet easy to access fashion… (remember Campbell’s wonderful idea of “reforms as experiments”?)

    Kudos to Australia!

      • Perhaps I’m a little fuzzy on present value, but I think I understand it as the value today of a future amount. Right? So I guess I’m stuck in this case because the future amount isn’t knowable, nor is the rate of accumulation of the future amount, which would give the interest rate or rate of return. So it’s hard sayin’ not knowin, as my friend matt used to say.

        It’s hard to tell quickly what they’re up to in this modellin’ business – presumably they’re looking at fund expense ratios – but when I see people suggesting that vendors are charging “excess costs” I’m concerned that there are mistaken – and perhaps unduly happy – assumptions about what constitutes an “excess cost” that can be eliminated to good effect. Folks don’t seem to get that as you drive down seller’s margins, they respond by aggregating into fewer larger vendors (eg oil industry 1990s), which recovers their pricing power and raises their margins – thereby blowing the shit out of the model assumptions. :)

        Whatever the case, since big projected government savings have a strong tendency to never occur, it would be a tough job to give them a present value.

        • Jim:

          In your original comment, you wrote, “t’ain’t saved a dang dime yet,” which suggests you’re assigning zero present value to future savings. In your next comment, you point out that the future savings are unknown. I agree that when the savings are unknown, this makes it more difficult and assumption-laden to calculate present value, but I don’t think that means we should set the present value to zero. That said, I’m speaking only in general terms here; I’ve not tried to evaluate the economics in this particular report.

        • The first comment is a rhetorical statement: models are one thing, reality is another thing. But if you want to take it literally, it is true right? At the moment, the net present value is zero. OK, it’s trivial of course because presumably it hasn’t been implemented, but no less true.

          The second comment speaks in mathematical terms the “net present value at time t” . For this instant, the net present value is zero. For any future instant, it’s unknown.

          I haven’t read the details either. And of course it’s possible that this model may bear substantial fruit. But as to the actual value of that fruit see my response to Daniel below.

        • I guess the proper terminology would be “Expected net present value” I think this is so ingrained in someone doing analysis of uncertain outcomes that they’re likely to drop the “expected” because it’s basically understood.

          Sure if you never implement a policy change, you’ll never see any value, but the assumption is that you will implement a policy change, and while the value is uncertain, it’s almost certainly NOT zero, and has expectation around 20B over 10 years.

          From the standpoint of the decision, it’s only the expectation that matters, since the actual outcome is unknowable at the time the decision will be made. Of course, you should only believe the expectation if you understand and feel that the model is “good”.

        • ” it’s almost certainly NOT zero”

          Do people understand allusion, allegory, simile and rhetoric anymore? :)

          I said the statement “T’ain’t nuthin but a dang model boys and girls, t’ain’t saved a dang dime yet.” is rhetorical. But it’s true, it hasn’t saved a dime yet. And it may never. right?

        • > But it’s true, it hasn’t saved a dime yet. And it may never. right?

          If and only if it never saves a dime has it not saved a dime yet. Present value is about moving money amounts between different times. So if it ever saves a dime at any point in the future, then today it’s saved the present value of that future dime.

        • I agree with Andrews take but wanted to add that it’s true that large predicted govt savings rarely pan out. on the other hand governments rarely publish any real data or models and seem to just make crap up, which is not the case here. And, this *isn’t* large. we are talking about say $20B/$3000B/10yrs = .067%/yr

        • Fair enough, but has the study discounted the present value of the savings? Assuming it hasn’t, which isn’t really clear but probably true, what are the $20B of savings accrued over 10 years actually worth in today’s $$

          Just roughly guessing that we’ll divide that savings over 10 years, then the savings @ t = 10 yr is $2B. This is an approximation, but good enough for getting a rough idea. OK, so assuming that inflation is the only factor affecting the value of our $2B, and that inflation hangs at 2%, how much do we need today to have $2B at t=10? My calculator says:

          If we start with $1.65B today we’ll wind up with $2B @ t=10. So in today’s $$ the $2B savings is worth about 18% less.

          But suppose we could invest our money in bonds that return interest today. Then it would take even fewer of today’s dollars to get $2B buying power @ t=10. If our bonds return a modest 2.25% yield on top of inflation (today’s 30yr rate is 2.23%), we’d only have to invest $1.32B to have $2B at t=10.

          But an astute investor could easily net 4% cash return with no more risk that T bills. So if you’re earning 4% today, to have $2B buying power ten years from now it costs only $1.12B.

          I don’t know the formula for how to average that over 10 years, but the discount is 44% overall in year ten, so figuring half of that in the average year, the quoted values are about 22% high to start with.

          So I guess, now that I’ve gone through this, I suspect part of the magic of inflated future savings is not discounting to the present value.

        • I don’t think this is the right analysis at all. The right analysis is more or less like there’s a bond hidden under the bed of some Australian grandmother that the government already owns, that pays the government $2B/yr but no one knows where it is to clip the coupons and mail it in.

          So they pay Jim Savage and company to go find it using some Bayesian search method. Maybe Jim and Co are say 5 or 6 people, each getting paid $100k/yr, and it takes them about 1 year to find this thing… So the cost is let’s say $500k, but after finding it the government gets “paid” $2B/yr for at least 10 years…

          So I fired up gnumeric, put 2 billion payments in a series of 10 years, discounted each one by 1.04^n for n from 1 to 10, added them up to find a little more than 16 billion in net present value, and then divided that by 500k to do the search for a rate of return of 3244358%

        • And this right here is my problem with all you scabs out there undercutting analysis work by working for a pittance… Come on Savage, you gotta be charging at least $150M for a return on investment of “only” 10815% or you’re basically just starving my kids ;-) …

          #hahaonlyserious

        • “The right analysis is more or less like there’s a bond hidden under the bed of some Australian grandmother that the government already owns, that pays the government $2B/yr”

          Not right at all. That’s why these things always go wrong. Hopeful people create models that favor their hopes.

          First, it’s not like a bond at all. A) no one knows if it exists or not; B) it doesn’t pay a fixed coupon; C) it can’t be traded and priced.

          Second, it doesn’t pay $2B a year. It *may* pay $20-50B over 10 years. And I now realize I overestimated the net present value because the basis is growing each year, so by far the largest single savings increment comes in the last year, when the net present value is lowest.

          Third, beyond the model I suggested above, and as I suggested to Andrew, the market will respond to the gov’s efforts to drive down fees by hiding them somewhere else or consolidating and raising prices or by changing their product mix to higher priced products.

        • I think it’s legitimate to question whether they’ve actually discovered 7 basis points in savings or whatever, but the analysis you’re doing is not right either.

          >First, it’s not like a bond at all. A) no one knows if it exists or not; B) it doesn’t pay a fixed coupon; C) it can’t be traded and priced.

          There can be bonds hidden under beds that we don’t know about… so that’s A, there are plenty of bonds that don’t pay fixed coupons, such as indexed bonds, so that’s B, and you could imagine a bond that the government owns that it’s not allowed to trade, but that still represents an obligation of some borrower to pay it a bunch of money… the fact that the government can’t trade it doesn’t mean it isn’t a bond, so that’s C.

          The fact is, there are costs associated with investing the Australian pension funds, and there are opportunities to alter the way that those investments work. Some of those opportunities would represent even larger costs (money flowing into the investment fund management’s pocket) and some of those opportunities represent lower costs (less money flowing into the mgmt more for the australian people).

          This report represents information that is relevant to the choices made, and points at expected increased returns under some assumptions (the assumptions are up for grabs).

          You seem to be thinking that “may pay 20-50 billion over 10 years” is like “it may pay 0, or 10 years from now it may pay a lump sum like 30 billion” but that’s not what we’re talking about at all. We’re talking about continuously saving a fluctuating amount around 5-10 basis points on cost throughout the next 10 years.

          So if you think of it as a biased coin flip that might come out 0 and might come out +30 billion in a lump sum, then yes you could say “hey there’s still a chance it’ll save zero” but this is more like “we’re going to pour a bunch of cash into a wind tunnel continuously for the next 10 years, and there’s a couple buckets at the end of the wind tunnel, and the wind tunnel operator is going to empty those buckets every hour and keep it… but we found a way to convince them to remove one of those buckets…” ***

          the chance that the bucket would have had zero cash flowing into it ever throughout the next 10 years is zero… removing the bucket is saving you money… how much is hard to quantify but that’s what Savage’s group tried to do.

          I’ll also say your point about the model assumptions getting violated is valid, there should be an analysis in which the effect of shifting 3T in assets to larger firms drives some kind of shift in the market and eliminates the economies of scale or reverses them… It wasn’t clear to me how much that entered into this model.

          *** for those following along the last few weeks, they built the money-wind-tunnel well before the money super-collider ;-)

        • “you seem to be thinking that “may pay 20-50 billion over 10 years” is like “it may pay 0, or 10 years from now it may pay a lump sum like 30 billion” but that’s not what we’re talking about at all. ”

          No, I’m not thinking that at all, I don’t know where you get that. I’m well aware that both the savings and the principle will compound.

          I doubt it will come out zero. I doubt it will produce the savings it claims either. I doubt that because even Stan can’t prevent a modeler from biasing a model, and a financial planning consultant has very strong incentives to provide a rosy forecast – the biggest one being that s/hell get paid long before the forecast savings are realized, and, with the good news spreading about the rosy forecast, may get paid several times before the real values come in.

        • >I doubt it will come out zero. I doubt it will produce the savings it claims either.

          Fair enough. It claims a wide-ish range, and in a Bayesian decision model, you shouldn’t be surprised if the outcome is different from the expected value, it’s just the quantity you use to make the decision because of the properties of the expectation operator. If it provides even 10B everyone still wins. The only serious problem would be if it actually costs money.

        • A small point:

          These aren’t real dollars, US dollars, you are talking about here. They are kangaroo dollars.

          True, some people say kangaroo dollars are almost as good as real dollars, but that assumes that the country you are talking about, Australia, is a real country. Sometimes I have my doubts. Pictures of things in Australia always show things as being right-side-up, when a quick look at a globe proves everything there is actually upside-down.

Leave a Reply to Jim - but not that one, this one! Cancel reply

Your email address will not be published. Required fields are marked *