Statistical Modeling, Causal Inference, and Social Science

In yesterday’s contest, Dalton asks:

Lance Armstrong isn’t even a GOAT. Did he cheat to get included on the list at the expense of Eddy Merckx?

But then Jrc points out:

Lance isn’t in for Cycling GOAT, he’s in for NGO-bracelet GOAT.

I’m pretty sure he didn’t juice the bracelets. Although now that I think about it… that might explain why they became so popular and after a while people had like 5 or 10 on at a time. So I guess since we already kicked out Erdos, it might be worth keeping him as an insurance policy in case we need some drugs to get through the tournament.

Ultimately, though, I’ll have to go with Daniel’s argument:

If Armstrong moves on – and ultimately wins – we’ll have to nervously wait 15 years to see if he recants his victory seminar.

We can’t have that, so it will have to be Sattouf, even though, as J points out, we can’t expect any sex from him, given that the seminar is in New York.

And next we have a bit of a Young Frankenstein contest. Actor vs. director. A man whose birth name didn’t really end in f vs. a man who’s two thousand years old. It’s your call.

Again, here’s the bracket and here are the rules.

Best comment yesterday came from Jan:

Now we have opportunity to see in the next round whether Julia is really that much better than Python!

But that doesn’t resolve anything! So to pick a winner we’ll have to go with Tom:

Python foresaw the replication crisis with their scientific method of proving someone is a witch but I fear that they would have to resort to talking about how they used to be funny. Springsteen could just bring a guitar and start playing – the only difficulty being that if you had something booked afterwards you might be a little late. Hmmm, not a particularly witty comment but still – onwards with Springsteen.

Today it’s the top-seeded person whose name ends in f, versus an unseeded GOAT. Sattouf is hilarious—but Lance does have a statistical principle named after him. So who should advance to the third round?

Again, here’s the bracket and here are the rules.

From Jeff:

If they meet in the semi-final the Japanese dude will eat Frank for lunch: All vs. Nothing at All.

Though it appears she also had a soft spot for hot dogs, if Julia makes it that far it would be a matchup of gourmet vs gourmand, which seems a better contest.

Today it’s an unseeded, but very funny, gang of Wits against the top seeded Person from New Jersey. What will it be: Holy Grail or Thunder Road?

Again, here’s the bracket and here are the rules.

For yesterday‘s contest, Jonathan gave a strong argument:

First New Yorker showdown, just to see who will be taking on Veronica Geng in the finals. All the other contestants are just for show. I’m going with Liebling, because Parker wasn’t even the best New Yorker writer of her generation, being edged out by Benchley. Liebling dominated his era. If it comes down to Liebling vs. Geng, we’ll just exhume Harold Ross and make him pick.

But we’re looking for a talker, not a writer, so I’ll have to go with Dzhaughn:

After the Seance, we were chatting about the inspiration for this tournament. I said I thought Bruno was just a minor intellectual swindler rather than a real threat. Dorothy replied:

I used to think Latour was just something on a Schwinn dealer’s list*, but that was before I saw Julia’s child Oscar wildly strong-arm Lance with an ephronedrine-filled syringe merrily down the Streep, past a sidewalk cafe where the turing Pele and big bejeweled #23, in Brooks’ Brothers suits, were yakking over Smirnoff Martinis, eating a pile of franks, caesar salads, and some weirder dishes. James was on the phone, taking the TV network to hell and back over “letting that degenerate George Karl off the hook” for some remark, when, from behind a bush, sudden as a python, out springs teen-aged Babe D.-Z, among others! That geng didn’t look like they were here to serenade us with arias from Yardbird, that jazz oprah about Parker! No, they were there to revolt–air their own grievances–and when he stood to object, Babe just shoved LeBron and all his LeBling back onto LaPlace where he sat: Oof!

A bit of recursion is usually a good plan.

For today it’s the French Chef vs. the Chairman of the Board. Frank’s got a less screechy voice, but Julia should be able to handle the refreshments. Any thoughts?

Again, here’s the bracket and here are the rules.

As Dalton wrote:

On one hand, Serena knows how to handle a racket. But Steve Martin knows how to make a racket with some strings stretched taught over a frame. Are you really gonna bet against the dude who went to toe-to-toe Kermit the Frog in racket making duel?

Today we have an unseeded eater vs. the second-seeded wit. That said, Liebling was very witty, at least in his writing. And Parker—I don’t know how she was as an eater, but she sure knew how to drink. As did Liebling. So the two are evenly matched.

Again, here’s the bracket and here are the rules.

Ron Berman writes:

I noticed you posted an anonymous email about our working paper on p-hacking and false discovery, but was a bit surprised that it references an early version of the paper.
We addressed the issues mentioned in the post more than two months ago in a version that has been available online since December 2018 (same link as above).

I wanted to send you a few thoughts on the post, with the hope that you will find them interesting and relevant to post on your blog as our reply, with the goal of informing your readers a little more about the paper.

These thoughts are presented in the separate section below.

The more recent analysis applies an MSE-optimal bandwidth selection procedure. Hence, we use only one single bandwidth for assessing the presence of a discontinuity at a specific level of confidence in an experiment.

Less importantly, the more recent analysis uses a triangular kernel and linear regression (though we also report a traditional logistic regression analysis result for transparency and robustness).
The results have not changed much, and have partially strengthened.

With regard to the RDD charts, the visual fit indeed might not be great. But we think the fit using the MSE-optimal window width is actually good.

The section below provides more details, and I hope you will find it relevant to post it on your blog.

We also of course would welcome any feedback you may have about the methods we are using in the paper, including the second part of the paper where we attempt to quantify the consequences of p-hacking on false discovery and foregone learning.

I am learning from every feedback we receive and am working to constantly improve the paper.

More details about blog post:

The comments by the anonymous letter writer are about an old version of the paper, and we have addressed them a few months ago.

Three main concerns were expressed: The choice of six possible discontinuities, the RDD window widths, and the RDD plot showing weak visual evidence of a discontinuity in stopping behavior based on the confidence level the experiment reaches.

1. Six hypotheses

We test six different hypotheses, each positing optional-stopping based on the p-value of the experiment at one of the three commonly used levels of significance/confidence in business and social science (90, 95 and 99%) for both positive and negative effects (3 X 2 = 6).
We view these as six distinct a-priori hypotheses, one each for a specific form of stopping behavior, not six tests of the same hypothesis.

2. RDD window width

The December 2018 version of the paper details an RDD analysis using an MSE-optimal bandwidth linear regression with a triangular kernel.
The results (and implications) haven’t changed dramatically using the more sophisticated approach, which relies on a single window for each RDD.

We fully report all the tables in the paper. This is what the results look like (Table 5 of the paper includes the details of the bandwidth sizes, number of observations etc):

The linear and the bias-corrected linear models use the “sophisticated” MSE-optimal method. We also report a logistic regression analysis with the same MSE-optimal window width for transparency and to show robustness.

All the effects are reported as marginal effects to allow easy comparison.

Not much has changed in results and the main conclusion about p-hacking remains the same: A sizable fraction of experiments exhibit credible evidence of stopping when the A/B test reaches 90% confidence for a positive effect, but not at the other levels of significance typically used and not for negative effects.

3. RDD plots

With respect to the RDD charts, the fit might indeed not look great visually. But what matters for the purpose of causal identification in such a quasi-experiment, in our opinion, is more the evidence of a discontinuity at the point of interest, rather than the overall data fit.

Here is the chart with the MSE-Optimal bandwidth around .895 confidence (presented as 90% to the experimenters) from the paper. Apart from the outlier at .89 confidence, we think the lines track the raw fractions rather well.

I wrote that earlier post in September, hence it was based on an earlier version of the article. It’s good to hear about the update.

We didn’t have much yesterday, so I went with this meta-style comment from Jesse:

I’m pulling for Kobayashi if only because the longer he’s in, the more often Andrew will have to justify describing him vs using his name. The thought of Andrew introducing the speaker as “and now, here’s that Japanese dude who won the hot dog eating contest” sounds awkward enough to prime us all for stress-eating, and who better to give us best practices/techniques?

I agree with Diana that there’s some underdog bias here, as in the real world there’d be no doubt that we’d want to hear Wilde. Indeed, if this were a serious contest we’d just have looked at the 64 names, picked Wilde right away, and ended it. But, for round 3, the Japanese dude who won the hot dog eating contest it is.

And today we have an unseeded GOAT vs. the fourth-seeded magician. Whaddya want, ground strokes or ironic jokes?

Again, here’s the bracket and here are the rules.

Someone who wishes to remain anonymous writes:

This paper [“p-Hacking and False Discovery in A/B Testing,” by Ron Berman, Leonid Pekelis, Aisling Scott, and Christophe Van den Bulte] ostensibly provides evidence of “p-hacking” in online experimentation (A/B testing) by looking at the decision to stop experiments right around thresholds for the platform presenting confidence that A beats B (which is just a transformation of the p-value).

It is a regression discontinuity design:

They even cite your paper [that must be this or this — ed.] against higher-order polynomials.

Indeed, the above regression discontinuity fits look pretty bad, as can be seen by imagining the scatterplots without those superimposed curves.

My correspondent continues:

The whole thing has forking paths and multiple comparisons all over it: they consider many different thresholds, then use both linear and quadratic fits with many different window sizes (not selected via standard methods), and then later parts of the paper focus only on the specifications that are the most significant (p less than 0.05, but p greater than 0.1).

Huh? Maybe he means “greater than 0.05, less than 0.1”? Whatever.

Anyway, he continues:

Example table (this is the one that looks best for them, others relegated to appendix):

So maybe an interesting tour of:
– How much optional stopping is there in industry? (Of course there is some.)
– Self-deception, ignorance, and incentive problems for social scientists
– Reasonable methods for regression discontinuity designs.

I’ve not read the paper in detail, so I’ll just repeat that I prefer to avoid the term “p-hacking,” which, to me, implies a purposeful gaming of the system. I prefer the expression “garden of forking paths” which allows for data-dependence in analysis, even without the researchers realizing it.

Also . . . just cos the analysis has statistical flaws, it doesn’t mean that the central claims of the paper in question are false. These could be true statements, even if they don’t quite have good enough data to prove them.

And one other point: There’s nothing at all wrong with data-dependent stopping rules. The problem is all in the use of p-values for making decisions. Use the data-dependent stopping rules, use Bayesian decision theory, and it all works out.

P.S. It’s been pointed out to me that the above-linked paper has been updated and improved since when I wrote the above post last September. Not all my comments above apply to the latest version of the paper.

Yesterday I was going to go with this argument from Ethan:

Now I’m morally bound to use the Erdos argument I said no one would see unless he made it to this round.

Andrew will take the speaker out to dinner, prove a theorem, publish it and earn an Erdos number of 1.

But then Jan pulled in with :

If you get Erdos, he will end up staying in your own place for the next n months, and him being dead, well, let’s say it is probably not going to be pleasant.

To be honest, I don’t even think I’d like a live Erdos staying in our apartment: from what I’ve read, the guy sounds a bit irritating, the kind of person who thinks he’s charming—an attribute that I find annoying.

Anyway, who cares about the Erdos number. What I really want is a good Wansink number. Recall what the notorious food researcher wrote:

Facebook, Twitter, Game of Thrones, Starbucks, spinning class . . . time management is tough when there’s so many other shiny alternatives that are more inviting than writing the background section or doing the analyses for a paper.

Yet most of us will never remember what we read or posted on Twitter or Facebook yesterday. In the meantime, this Turkish woman’s resume will always have the five papers below.

Coauthorship is forever. Those of us with a low Wansink number will live forever in the scientific literature.

And today’s match features an unseeded eater vs. the top-seeded wit. Doesn’t seem like much of a contest for a seminar speaker, but . . . let’s see what arguments you come up with!

Again, here’s the bracket and here are the rules.