Debate over claims of importance of spending on Obamacare advertising

Jerrod Anderson points to this post by Paul Shafer, Erika Fowler, Laura Baum, and Sarah Gollust, “Advertising cutbacks reduce Marketplace information-seeking behavior: Lessons from Kentucky for 2018.”

Anderson expresses skepticism about this claim.

I’ll first summarize the claims of Shafer et al. and then get to Anderson’s criticism.

Shafer et al. write:

The Trump administration announced Thursday that it was cutting spending on advertising for the 2018 Marketplace open enrollment period from $100 to $10 million. Empirical work can inform our expectations for its impact, assuming these cuts are implemented. . . .

Kentucky—an early success story under the ACA—sponsored a robust multimedia campaign to create awareness about its state-based marketplace, known as kynect, to educate its residents about the opportunity to gain coverage. However, after the 2015 gubernatorial election, the Bevin administration declined to renew the advertising contract for kynect and directed all pending advertisements to be canceled with approximately six weeks remaining in the 2016 open enrollment period. The reduction in advertising during open enrollment gives us precisely the rare leverage needed to assess the influence of advertising using real-world data.

We obtained advertising and Marketplace data in Kentucky to identify whether a dose-response relationship exists between weekly advertising volume and information-seeking behavior. . . .

Each additional kynect ad per week during open enrollment was associated with an additional 7,973 page views (P=.001), 390 visits (P=.003), and 388 unique visitors (P<.001) to the kynect web site per week. Based on the average number of ads per week during the first two open enrollment periods, our estimates imply that there would have been more than 450,000 fewer page views, 20,000 fewer visits, and 20,000 fewer unique visitors per week during open enrollment without the television campaign. . . .

But Anderson is concerned that the changes attributed to advertising are explainable more simply as artifacts arising from timing of the treatment variation:

I’ve seen this linked to from a few other sites so I thought you might want to comment.

The authors run a linear regression with the unique visitors, site visits, and calls as the dependent variables and the number of weekly ads by the state exchange (among other “controls”) as the independent variable of interest. Aside from the usual problems of hypothesis testing and forking paths and beyond the concerns that the effect is probably nonlinear enough to make a linear regression inappropriate, the co-movements of the unique visitors and the number of weekly ads just doesn’t seem that convincing. Notice the very beginning of the study period in the graph, there are 160,000 unique visitors (which is what we’re really concerned about). That number is cut in half after the first week (probably because there were several people just curious about the website) and this outlier may have a significant effect on the results. If you look at the variance in weekly advertising outside of the open enrollment periods, the number of unique visitors barely changes and sometimes the big changes precede any increase in advertising. My sense is that the beginning and ending of open enrollment are the main drivers of visits and visitors and those just happen to coincide with large changes in advertising.

6 thoughts on “Debate over claims of importance of spending on Obamacare advertising

  1. I have a really fun and surprisingly relevant anecdote.

    In 2014, I was working with the office of the Governor of Illinois to help them assess the impact of advertising spending for the Illinois ACA exchange on information-seeking behaviour (inbound calls; website visits). As noted above, there’s a pretty enormous endogeneity problem: the media buyer timed most advertisements to the points of the open enrollment period in which people would be seeking information: at the beginning and end of the period. So you get incredible co-movement between ad buys and inbound traffic.

    I was putting together a time-series model (something a bit like Google’s causalimpact VAR) and noticed that when I left-joined the daily visits/calls data to the ad buys, there were a bunch of dates in which there were no ad-buys at all. We contacted the media buyer to ask what was the problem. It turns out that mid-way through the open enrollment period, the governor’s office had entered contract renegotiations with the media buyer, and no ads were run during the period. They were very apologetic for not having given me a complete dataset.

    Hey presto– they’d run a fairly rough natural experiment! Using the dates of contract renegotiation to instrument for ad buys, we got pretty good estimates of the causal impact of advertising on inbound information requests. The takeaway figure was that the ads run on Spanish-language channels had multiples of the impact per dollar spent of the ads on the English-language channels (30% of Illinois’ uninsured are Latino).

    • Note that the effect for subgroups, pace Savage above, could well exist, but simply be lost in the aggregate, or be noise.

  2. And here I thought a relative handful of Facebook ads subverted the election. Ads are so powerful and effective, never a way of money and they always engage!

  3. Hi Andrew-
    I have a paper on Medicare Advantage advertising that is relevant and you might be interested in: https://papers.ssrn.com/abstract=2789904. I’m aware that MA and ACA are not exactly the same market, but we might expect advertising to work under similar theories. I find evidence of very very small ad effects (most economically unimportant). I find that the small cannot be attributed to competitive effects of advertising canceling each other out and find nothing that indicates a longer-term effect of advertising.
    Best,
    Brad

Leave a Reply to Jonathan (another one) Cancel reply

Your email address will not be published. Required fields are marked *