Statistical Modeling, Causal Inference, and Social Science

This post was originally titled, “Asleep at the wheel: Junk science gets promoted by Bill Gates, NPR, Ted, University of California, Google, etc.,” but I decided the above quote had better literary value.

We’ve had a few posts now about discredited sleep scientist Matthew Walker; see here and here, and here, with the disappointing but unsurprising followup that his employer, the University of California, just doesn’t seem to care about his research misconduct. According to his website, Walker is employed by Google as well.

Heart attacks

Also disappointingly but unsurprisingly, further exploration of Walker’s work reveals further misrepresentations of research.

Markus Loecher shared this investigation of some attention-grabbing claims made in the celebrated Ted talk, “Sleep is your superpower.”

Here’s Walker:

I could tell you about sleep loss and your cardiovascular system, and that all it takes is one hour. Because there is a global experiment performed on 1.6 billion people across 70 countries twice a year, and it’s called daylight saving time. Now, in the spring, when we lose one hour of sleep, we see a subsequent 24-percent increase in heart attacks that following day. In the autumn, when we gain an hour of sleep, we see a 21-percent reduction in heart attacks. Isn’t that incredible? And you see exactly the same profile for car crashes, road traffic accidents, even suicide rates.

“Isn’t that incredible?”, indeed. This reminds me of the principle that, if something is being advertised as “incredible,” it probably is.

Loecher decided to look into the above claim:

I [Loecher] tend to be sensitive to gross exaggerations disguised as “scientific findings” and upon hearing of such a ridiculously large effect of a one-day-one-hour sleep disturbance, all of my alarm bells went up!

Initially I was super excited about the suggested sample size of 1.6 billion people and wanted to find out how exactly such an incredible data set could possibly have been gathered. Upon my inquiry, Matthew was kind enough to point me to the paper, which was the basis for the rather outrageous claims from above. Luckily, it is an open access article in the openheart Journal from 2014.

Imagine my grave disappointment to find out that the sample was limited to 3 years in the state of Michigan and had just 31 cases per day! On page 4 you find Table 1 which contains the quoted 24% increase and 21% decrease expressed as relative risk (multipliers 1.24 and 0.79, respectively):

Loecher notes the obvious multiple comparisons issues.

But what I want to focus on is the manipulation or incompetence (recall Clarke’s Law).

To start with, here’s the summary from the above-linked article:

Now, we could argue about whether the data really show that daylight savings time “impacts the timing” of acute myocardial infarction—arguably, the data here are consistent with no effect on timing at all! But let’s set that aside and focus on the other point of their summary: daylight savings time “does not influence the overall incidence of this disease.”

This completely contradicts Walker’s theme of sleep deprivation being dangerous. It did not influence the overall incidence of the disease!

Presumably Walker realized this: even if he didn’t read the whole article, he must have read the abstract, at least to pull out those 24% and 21% numbers. (Or maybe Walker’s research assistant did it, but no matter. If Walker gets credit for the book and the Ted talk, he also gets blame for the errors and misrepresentations that he puts out under his name.)

So . . . he read a paper claiming that, at most, daylight time is associated with some time-shifting of heart attacks, and he misrepresents that as being associated with an increase.

Also, he says is “a global experiment performed on 1.6 billion people,” but he’s reporting results on one U.S. state. He must have realized that too, no, that this was not a N = 1.6 billion study???

But wait, there’s more. We switch to daylight time 2am on Sunday. So you might expect the largest effects to occur on Sunday—that day with the sleep deprivation. Or maybe Monday, the first day back at work. All sorts of things are possible. The point is that that, by saying it as “that following day,” Walker is hiding the choice. If it’d been Sunday, it would’ve been “the very day of,” etc. And then when he talks about autumn, he doesn’t say “that following day,” just leaving the (false) impression that it’s the same pattern both seasons.

Tuesday, huh?

I also wonder about Walker’s other claim, that at the switch to daylight or standard time “you see exactly the same profile for car crashes, road traffic accidents, even suicide rates.”

Exactly the same, huh? I’ll believe it when I see the data, and not before.

Testicles

Loecher also looks into Walker’s claim that “Men who sleep five hours a night have significantly smaller testicles than those who sleep seven hours or more.” There seems to be no good evidence for that one either.

Ted and Edge and all the rest

I hate the whole Ted talk, Edge foundation, Great Man model of science. It can destroy people. As I wrote last year:

Don’t ever think you’re too good for your data. . . .

Which reality do we care about? The scientific reality of measurement and data, or the social reality that a Harvard [or University of California] professor can get caught falsifying data and still be presented as an authority on science and philosophy. Ultimately, both realities matter. But let’s not confuse them. Let’s not confuse social power with scientific evidence. Remember Lysenko. . . .

OK, why am I picking on these guys? Marc Hauser and the Edge foundation: are these not the deadest of dead horses? But remember what they say about beating a dead horse. The larger issue—a smug pseudo-humanistic contempt for scientific measurement, along with an attitude that money plus fame = truth—that’s still out there.

An always-relevant quote

From Dan Davies: Good ideas do not need lots of lies told about them in order to gain public acceptance.

A message to Bill Gates, NPR, Ted, University of California, Google, etc.

It’s not your fault that you got scammed. I mean, sure, it’s kind of your fault for not checking, but that’s easy to say after the fact. Anyone can get scammed. I’ve been scammed! My political scientist colleague Don Green got scammed! Harvard got scammed by that disgraced primatologist. Cornell got scammed . . . George Schultz got scammed . . . maybe you’ve heard about that one. People get scammed. Scammers scam people, that’s what they do.

You got scammed. That’s in the past, now.

The question is: Are you gonna continue to let yourself get scammed?

I’ll break that down into 2 questions:

1. Do you now, at last, realize you’ve been scammed? (If not, what would it take? An actual manipulated graph?? No, I guess not; we already have one of those!)

2. If your answer to the first question is Yes, then are you gonna decide that it’s worth your while to continue to get scammed, because the cost in effort and bad publicity is worse than the cost of continuing to promote this stuff?

If the answer to question 1 is No, that’s just sad, that people could see all this evidence and still not get the point.

If the answer to question 1 is Yes and the answer to question 2 is No, that’s even sadder.

A quick google search appears to reveal six separate appearances by Walker on NPR, with the most recent being this June, several months after Guzey’s takedown of Why We Sleep. But I’m guessing that, once NPR had Walker on once or twice, he became a known quantity for them, so they just put him in the Expert category.

Ted? They’ve had iffy talks before, I guess it comes with the territory. They’re probably focusing on scheduling and promoting new talks, not on problems with talks they’ve already posted.

Bill Gates? He endorsed Walker’s book and now he’s moved on. Gates probably doesn’t care about an endorsement that’s sitting in the past.

The University of California? They know about the problems with Walker’s work but they’ve carefully looked away. I think they’re basically Yes on question 1 and No on question 2, except that they’ve tried really hard to avoid answering question 1. At some level, they must know they’ve been scammed, but as long as they avoid looking at Walker’s work carefully (even to the extent of carefully reading a few blog posts), they can maintain a facade of uncertainty.

Google? I have no idea. I don’t know what Walker does for Google. Maybe he’s doing great work for them. Yes, he has a problem with exaggerating research claims in publications and public talks, but maybe he does excellent work when the lights of publicity are not shining. In that case, he’s not scamming Google at all.

Also, Walker could well be scamming himself. I’m not trying to paint him as some cackling villain here. I could well imagine he’s a true believer in the healing power of sleep, and that when he misrepresents the evidence, in his view that’s just because he doesn’t have all the data at hand. Sure, they didn’t really have data on 1.6 billion people, but if they did, it would undoubtedly confirm his views. He has a direct line to the truth, and he’d be remiss if he didn’t shout it from the treetops.

The trouble is, people who think they have a direct line to the truth, often’t don’t. Recall the above quote from Dan Davies. Tony Blair probably thought those WMDs were real too, at some point—or, if they weren’t, they could’ve been, right? And the war was a good idea anyway, right? Etc. To return to sleep studies: the data support the theory, and once you believe the theory, you don’t need the data anymore.

As always, I’d have no problem if this guy were to just straight-up give inspirational talks and write inspirational books. He could just say that he personally believes in the importance of sleep, and his broad reading and experience has lead him to this conclusion. It’s when he misrepresents data, that’s where I have the problem (even though the University of California doesn’t seem to mind).

Palko points to this post by Jeffrey Funk, “What’s Behind Technological Hype?”

I’ll quote extensively from Funk’s post, but first I want to make a more general point about the burden of proof in scientific discussions.

What happens is that a researcher or team of researchers makes a strong claim that is not well supported by evidence. But the claim gets published, perhaps in a prestigious journal such as the Journal of Theoretical Biology or PNAS or Lancet or the American Economic Review or Psychological Science. The authors may well be completely sincere in their belief that they’ve demonstrated something important, but their data don’t really back up their claim. Later on, the claims are criticized: outside researchers look carefully at the published material and point out that the evidence isn’t really there. Fine. The problem arises when the critics are then held to a higher standard: it’s not enough for them to point out that the original paper did not offer strong evidence for its striking claim; the critics are asked to (impossibly) prove that the claimed effect cannot possibly be true.

It’s a sort of Cheshire Cat phenomenon: Original researchers propose a striking and noteworthy (i.e., not completely obvious) idea, which is published and given major publicity based on purportedly strong statistical and experimental evidence. The strong evidence turns out not to be there, but—like the smile of the Cheshire cat—the claim remains even after the evidence has disappeared.

This is related to what we’ve called the “research incumbency advantage” (the widespread attitude that a published claim is considered true unless conclusively proved otherwise), and the “time-reversal heuristic” (my suggestion to suppose that the counter-argument or failed replication came first, with the splashy study following after).

Now to Funk’s post on technological hype:

Start-up losses are mounting and innovation is slowing. . . . The large losses are easily explained: extreme levels of hype about new technologies, and too many investors willing to believe it. . . . The media, with help from the financial sector, supports the hype, offering logical reasons for the [stock] price increases and creating a narrative that encourages still more increases. . . .

The [recent] narrative began with Ray Kurzweil’s 2005 book, The Singularity is Near, and has expanded with bestsellers such as Erik Brynjolfsson and Andrew McAfee’s Race Against the Machine (2012), Peter Diamandis and Steven Kotler’s Abundance (2012), and Martin Ford’s The Rise of the Robots (2015). Supported by soaring venture capitalist investments and a rising stock market, the world described in these books is one of rapid and disruptive technological change that will soon lead to great prosperity and perhaps massive unemployment. The media has amplified this message even as evidence of rising productivity or unemployment has yet to emerge.

Here I [Funk] discuss economic data showing that many highly touted new technologies are seriously over-hyped, a phenomenon driven by online news and the professional incentives of those involved in promoting innovation and entrepreneurship. This hype comes at a cost—not only in the form of record losses by start-ups, but in their inability to pursue alternative designs and find more productive and profitable opportunities . . .

These indicators are widely ignored, in part because we are distracted by information appearing to carry a more positive message. The number of patent applications and patent awards has increased about sixfold since 1984, and over the past 10 years the number of scientific papers has doubled. The stock market has tripled in value since 2008. Investments by US venture capitalists have risen about sixfold since 2001 . . . Such upward trends are often used to hype the economic potential of new technologies, but in fact rising patent activity, scientific publication, stock market value, and venture capital investment are all poor indicators of innovativeness.

One reason they are poor indicators is that they don’t consider the record-high losses for start-ups, the lack of innovations for large sectors of the economy such as housing, and the small range of technologies being successfully commercialized by either start-ups or existing firms. . . .

Funk then talks about the sources of hype:

For more recent technologies such as artificial intelligence, a major source of hype is the tendency of tech analysts to extrapolate from one or two highly valued yet unprofitable start-ups to total disruptions of entire sectors. For example, in its report Artificial Intelligence: The Next Digital Frontier? the McKinsey Global Institute extrapolated from the purported success of two early AI start-ups, DeepMind and Nest Labs, both subsidiaries of Alphabet (Google’s parent company), to a 10% reduction in total energy usage in the United Kingdom and other countries. However, other evidence for these purported energy reductions in data centers and homes are nowhere to be found, and the start-ups are currently a long way from profitability. Alphabet reported losses of approximately $580 million in 2017 for DeepMind and $569 million in 2018 for Nest Labs. . . .

Hype and its amplification come from many quarters: not only the financial community but also entrepreneurs, venture capitalists, consultants, scientists, engineers, and universities. . . .

Ya think??

Funk continues:

Online tech-hyping articles are now driven by the same dynamics as fake news. Journalists, bloggers, and websites prioritize page views and therefore say more positive things to attract viewers, while social media works as an amplifier. Journalists become “content marketers,” often hired by start-ups and universities to promote new technologies. Entrepreneurs, venture capitalists, university public relation offices, entrepreneurship programs, and professors who benefit from the promotion of new technologies all end up sharing an interest in increasing the level of hype. . . .

And this connects to the point I made at the beginning of this post. Once a hyped idea gets out there, it’s the default, and merely removing the evidence in favor is not enough. Mars One, Hyperloop, etc.: sure, eventually they fade, but in the meantime they suck up media attention and $$$, in part because they become the default, and the burden of proof is on the skeptics.

Selection bias in tech and science reporting

One other thing: the remark that journalists etc. “say more positive things to attract viewers” reminds me of what I’ve written about selection bias in science reporting (see also here). Lots of science reporters want to do the right thing, and, yes, they want clicks and they want to report positive stories—I too would be much more interested to read or write about a cure for cancer than about some bogus bit of noise mining—and these reporters will steer away from junk science. But here’s where the selection bias comes in: other, less savvy or selective or scrupulous reporters will jump in and hype the junk. So, with rare exceptions (some studies are so bad and so juicy that they just beg to be publicly debunked), the bad studies get promoted by the clueless journalists, and the negative reports don’t get written.

My point here is that selection bias can give us a sort of Gresham effect, even without any journalists knowingly hyping anything of low quality.

Gregory Hunter writes:

The following article made it to the national news in Canada this week.

I [Hunter] read it and was fairly appalled by their statistical methods. It seems that they went looking for a particular result in Canadian birthrate data, and then arranged to find it. Figure 1 looks like random variation to me. I don’t know if it warrants mention in your blog, but it did get into the British Medical Journal.

That’s too bad about it being in the British Medical Journal. Lancet, sure, they’re notorious for publishing politically motivated clickbait. But I thought BMJ was more serious. I guess anyone can make mistakes.

Anyway, getting to the statistics . . . the article is called “Outcome of the 2016 United States presidential election and the subsequent sex ratio at birth in Canada: an ecological study,” and its results are a mess of forking paths:

We hypothesised that the unexpected outcome of the 2016 US presidential election may have been a societal stressor for liberal-leaning populations and thereby precipitated such an effect on the sex ratio in Canada. . . . In the 12 months following the election, the lowest sex ratio occurred in March 2017 (4 months post election). Compared with the preceding months, the sex ratio was lower in the 5 months from March to July 2017 (p=0.02) during which time it was rising (p=0.01), reflecting recovery from the nadir. Both effects were seen in liberal-leaning regions of Ontario (lower sex ratio (p=0.006) and recovery (p=0.002) in March–July 2017) but not in conservative-leaning areas (p=0.12 and p=0.49, respectively).

In addition to forking paths, we also see the statistical fallacy of comparing significant to non-significant.

To their credit, the authors show the data:

As is often the case, if you look at the data without all those lines, you see something that looks like a bunch of numbers with no clear pattern.

The claims made in this article do not represent innumeracy on the level of saying that the probability of a tied election is 10^-90 (which is off by a factor of 10^83), and it’s not innumeracy on the level of that TV commenter and newspaper editor who said that Mike Bloomberg spent a million dollars on each voter (off by a factor of 10^6), but it’s still wrong.

Just to get a baseline here: There were 146,000 births in Ontario last year. 146,000/12 = 12,000 (approximately). So, just from pure chance, we’d expect the monthly proportion of girl births to vary with a standard deviation of 0.5/sqrt(12000) = 0.005. For example if the baseline rate is 48.5% girls, it could jump to 48.0% or 49.0% from month to month. The paper in question reports sex ratio, which is (1-p)/p, so 0.480, 0.498, 0.490 convert to sex ratios of 1.08, 1.06, and 1.04. Or, if you want to do +/-2 standard deviations, you’d expect to see sex ratios varying from roughly 1.10 to 1.02, which is indeed what we see in the top figure above. (The lower figures are each based on less data so of course they’re more variable.) Any real effects on sex ratio will be tiny compared to this variation in the data (see here for discussion of this general point).

In short: this study was dead on arrival. But the authors fooled themselves, and the reviewers, with a blizzard of p-values. As low as 0.002!

So, let me repeat:

– Just cos you have a statistically significant comparison, that doesn’t necessarily mean you’ve discovered anything at all about the world.

– Just cos you have causal identification and a statistically significant comparison, that doesn’t necessarily mean you’ve discovered anything at all about the world.

– Just cos you have honesty, transparency, causal identification, and a statistically significant comparison, that doesn’t necessarily mean you’ve discovered anything at all about the world.

– Just cos you have honesty, transparency, causal identification, a statistically significant comparison, a clear moral purpose, and publication in a top journal, that doesn’t necessarily mean you’ve discovered anything at all about the world.

Sorry, but that’s the way it is. You’d think everyone would’ve learned this—it’s been nearly a decade since that ESP paper was published—but I guess not. The old ways of thinking are sticky. Sticky sticky sticky.

Again, no special criticism on the authors of this new paper. I assume they’re just doing what they were trained to do, and now what they’re rewarded to do. Don’t hate the player etc.