How often is there a political candidate such as Vivek Ramaswamy who is so much stronger in online polls than telephone polls?

Palko points to this news article, “The mystery of Vivek Ramaswamy’s rapid rise in the polls,” which states:

Ramaswamy’s strength comes almost entirely from polls conducted over the internet, according to a POLITICO analysis. In internet surveys over the past month — the vast majority of which are conducted among panels of people who sign up ahead of time to complete polls, often for financial incentives — Ramaswamy earns an average of 7.8 percent, a clear third behind Trump and DeSantis.

In polls conducted mostly or partially over the telephone, in which people are contacted randomly, not only does Ramaswamy lag his average score — he’s way back in seventh place, at just 2.6 percent.

There’s no singular, obvious explanation for the disparity, but there are some leading theories for it, namely the demographic characteristics and internet literacy of Ramaswamy’s supporters, along with the complications of an overly white audience trying to pronounce the name of a son of immigrants from India over the phone.”

And then, in order for a respondent to choose Ramaswamy in a phone poll, he or she will have to repeat the name back to the interviewer. And the national Republican electorate is definitely older and whiter than the country as a whole: In a recent New York Times/Siena College poll, more than 80 percent of likely GOP primary voters were white, and 38 percent were 65 or older.

‘When your candidate is named Vivek Ramaswamy,’ said one Republican pollster, granted anonymity to discuss the polling dynamics candidate, ‘that’s like DEFCON 1 for confusion and mispronunciation.’

Palko writes:

Keeping in mind that the “surge” was never big (maxed out at 10% and has been flat since), we’re talking about fairly small numbers in absolute terms, here are some questions:

1. How much do we normally expect phone and online to agree?

2. Ramaswamy generally scores around 3 times higher online than with phone. Have we seen that magnitude before?

3. How about a difficult name bias. Have we seen that before? How about Buttigieg, for instance? Did a foreign-sounding name hurt Obama in early polls?

4. Is the difference in demographics great enough to explain the difference? Aren’t things like gender and age normally reweighted?

5. Are there other explanations we should consider?

I don’t have any answers here, just one thought which is that it’s early in the campaign (I guess I should call it the pre-campaign, given that the primary elections haven’t started yet), and so perhaps journalists are reasoning that, even if this candidate is not very popular among voters, his active internet presence makes him a reasonable dark-horse candidate looking forward. An elite taste now but could perhaps spread to the non-political-junkies in the future? Paradoxically, the fact that Ramaswamy has this strong online support despite his extreme political stances could be taken as a potential sign of strength? I don’t know.

“Hot hand”: The controversy that shouldn’t be. And thinking more about what makes something into a controversy:

I was involved in a recent email discussion, leading to this summary:

There is no theoretical or empirical reason for the hot hand to be controversial. The only good reason for there being a controversy is that the mistaken paper by Gilovich et al. appeared first. At this point we should give Gilovich et al. credit for bringing up the hot hand as a subject of study and accept that they were wrong in their theory, empirics, and conclusions, and we can all move on. There is no shame in this for Gilovich et al. We all make mistakes, and what’s important is not the personalities but the research that leads to understanding, often through tortuous routes.

“No theoretical reason”: see discussion here, for example.

“No empirical reason”: see here and lots more in the recent literature.

“The only good reason . . . appeared first”: Beware the research incumbency rule.

More generally, what makes something a controversy? I’m not quite sure, but I think the news media play a big part. We talked about this recently in the context of the always-popular UFOs-as-space-aliens theory, which used to be considered a joke in polite company but now seems to have reached the level of controversy.

I don’t have anything systematic to say about all this right now, but the general topic seems very worthy of study.

Their signal-to-noise ratio was low, so they decided to do a specification search, use a one-tailed test, and go with a p-value of 0.1.

Adam Zelizer writes:

I saw your post about the underpowered COVID survey experiment on the blog and wondered if you’ve seen this paper, “Counter-stereotypical Messaging and Partisan Cues: Moving the Needle on Vaccines in a Polarized U.S.” It is written by a strong team of economists and political scientists and finds large positive effects of Trump pro-vaccine messaging on vaccine uptake.

They find large positive effects of the messaging (administered through Youtube ads) on the number of vaccines administered at the county level—over 100 new vaccinations in treated counties—but only after changing their specification from the prespecified one in the PAP. The p-value from the main modified specification is only 0.097, from a one-tailed test, and the effect size from the modified specification is 10 times larger than what they get from the pre-specified model. The prespecified model finds that showing the Trump advertisement increased the number of vaccines administered in the average treated county by 10; the specification in the paper, and reported in the abstract, estimates 103 more vaccines. So moving from the specification in the PAP to the one in the paper doesn’t just improve precision, but it dramatically increases the estimated treatment effect. A good example of suppression effects.

They explain their logic for using the modified specification, but it smells like the garden of forking paths.

Here’s a snippet from the article:

I don’t have much to say about the forking paths except to give my usual advice to fit all reasonable specifications and use a hierarchical model, or at the very least do a multiverse analysis. No reason to think that the effect of this treatment should be zero, and if you really care about effect size you want to avoid obvious sources of bias such as model selection.

The above bit about one-tailed tests reflects a common misunderstanding in social science. As I’ll keep saying until my lips bleed, effects are never zero. They’re large in some settings, small in others, sometimes positive, sometimes negative. From the perspective of the researchers, the idea of the hypothesis test is to give convincing evidence that the treatment truly has a positive average effect. That’s fine, and it’s addressed directly through estimation: the uncertainty interval gives you a sense of what the data can tell you here.

When they say they’re doing a one-tailed test and they’re cool with a p-value of 0.1 (that would be 0.2 when following the standard approach) because they have “low signal-to-noise ratios” . . . that’s just wack. Low signal-to-noise ratio implies high uncertainty in your conclusions. High uncertainty is fine! You can still recommend this policy be done in the midst of this uncertainty. After all, policymakers have to do something. To me, this one-sided testing and p-value thresholding thing just seems to be missing the point, in that it’s trying to squeeze out an expression of near-certainty from data that don’t admit such an interpretation.

P.S. I do not write this sort of post out of any sort of animosity toward the authors or toward their topic of research. I write about these methods issues because I care. Policy is important. I don’t think it is good for policy for researchers to use statistical methods that lead to overconfidence and inappropriate impressions of certainty or near-certainty. The goal of a statistical analysis should not be to attain statistical significance or to otherwise reach some sort of success point. It should be to learn what we can from our data and model, and to also get a sense of what we don’t know..

How to code and impute income in studies of opinion polls?

Nate Cohn asks:

What’s your preferred way to handle income in a regression when income categories are inconsistent across several combined survey datasets? Am I best off just handling this with multiple categorical variables? Can I safely create a continuous variable?

My reply:

I thought a lot about this issue when writing Red Sate Blue State. My preferred strategy is to use a variable that we could treat as continuous. For example when working with ANES data I was using income categories 1,2,3,4,5 which corresponded to income categories 1-16th percentile, 16-33rd, 34-66th, 67-95th, and 96-100th. If you have different surveys with different categories, you could use some somewhat consistent scaling, for example one survey you might code as 1,3,5,7 and another might be coded as 2,4,6,8. I expect that other people would disagree with this advice but this the sort of thing that I was doing. I’m not so much worried about the scale being imperfect or nonlinear. But if you have a non-monotonic relation, you’ll have to be more careful.

Cohn responds:

Two other thoughts for consideration:

— I am concerned about non-monotonicity. At least in this compilation of 2020 data, the Democrats do best among rich and poor, and sag in the middle. It seems even more extreme when we get into the highest/lowest income strata, ala ANES. I’m not sure this survives controls—it seems like there’s basically no income effect after controls—but I’m hesitant to squelch a possible non-monotonic effect that I haven’t ruled out.

—I’m also curious for your thoughts on a related case. Suppose that (a) dataset includes surveys that sometimes asked about income and sometimes did not ask about income, (b) we’re interested in many demographic covariates, besides income, and; (c) we’d otherwise clearly specify the interaction between income and the other variables. The missing income data creates several challenges. What should we do?

I can imagine some hacky solutions to the NA data problem outright removing observations (say, set all NA income to 1 and interact our continuous income variable with whether we have actual income data), but if we interact other variables with the NA income data there are lots of cases (say, MRP where the population strata specifies income for full pop, not in proportion to survey coverage) where we’d risk losing much of the power gleaned from other surveys about the other demographic covariates. What should we do here?

My quick recommendation is to fit a model with two stages, first predicting income given your other covariates, then predicting your outcome of interest (issue attitude, vote preference, whatever) given income and the other covariates. You can fit the two models simultaneously in one Stan program. I guess then you will want some continuous coding for income (could be something like sqrt(income) with income topcoded at $300K) along with a possibly non-monotonic model at the second level.

Minor-league Stats Predict Major-league Performance, Sarah Palin, and Some Differences Between Baseball and Politics

In politics, as in baseball, hot prospects from the minors can have trouble handling big-league pitching.

Right after Sarah Palin was chosen as the Republican nominee for vice president in 2008, my friend Ubs, who grew up in Alaska and follows politics closely, wrote the following:

Palin would probably be a pretty good president. . . . She is fantastically popular. Her percentage approval ratings have reached the 90s. Even now, with a minor nepotism scandal going on, she’s still about 80%. . . . How does one do that? You might get 60% or 70% who are rabidly enthusiastic in their love and support, but you’re also going to get a solid core of opposition who hate you with nearly as much passion. The way you get to 90% is by being boringly competent while remaining inoffensive to people all across the political spectrum.

Ubs gives a long discussion of Alaska’s unique politics and then writes:

Palin’s magic formula for success has been simply to ignore partisan crap and get down to the boring business of fixing up a broken government. . . . It’s not a very exciting answer, but it is, I think, why she gets high approval ratings — because all the Democrats, Libertarians, and centrists appreciate that she’s doing a good job on the boring non-partisan stuff that everyone agrees on and she isn’t pissing them off by doing anything on the partisan stuff where they disagree.

Hey–I bet you never thought you’d see the words “boringly competent,” “inoffensive,” and “Sarah Palin” in the same sentence!

Prediction and extrapolation

OK, so what’s the big deal? Palin got a reputation as a competent nonpartisan governor but when she hit the big stage she shifted to hyper-partisanship. The contrast is interesting to me because it suggests a failure of extrapolation.

Now let’s move to baseball. One of the big findings of baseball statistics guru Bill James is that minor-league statistics, when correctly adjusted, predict major-league performance. James is working through a three-step process: (1) naive trust in minor league stats, (2) a recognition that raw minor league stats are misleading, (3) a statistical adjustment process, by which you realize that there really is a lot of information there, if you know how to use it.

For a political analogy, consider Scott Brown. When he was running for the Senate last year, political scientist Boris Shor analyzed his political ideology. The question was, how would he vote in the Senate if he were elected? Boris wrote:

We have evidence from multiple sources. The Boston Globe, in its editorial endorsing Coakley, called Brown “in the mode of the national GOP.” Liberal bloggers have tried to tie him to the Tea Party movement, making him out to be very conservative. Chuck Schumer called him “far-right.”

In 2002, he filled out a Votesmart survey on his policy positions in the context of running for the State Senate. Looking through the answers doesn’t reveal too much beyond that he is a pro-choice, anti-tax, pro-gun Republican. His interest group ratings are all over the map. . . .

All in all, a very confusing assessment, and quite imprecise. So how do we compare Brown to other state legislators, or more generally to other politicians across the country?

My [Boris’s] research, along with Princeton’s Nolan McCarty, allows us to make precisely these comparisons. Essentially, I use the entirety of state legislative voting records across the country, and I make them comparable by calibrating them through Project Votesmart’s candidate surveys.

By doing so, I can estimate Brown’s ideological score very precisely. It turns out that his score is -0.17, compared with her score of 0.02. Liberals have lower scores; conservatives higher ones. Brown’s score puts him at the 34th percentile of his party in Massachusetts over the 1995-2006 time period. In other words, two thirds of other Massachusetts Republican state legislators were more conservative than he was. This is evidence for my [Boris’s] claim that he’s a liberal even in his own party. What’s remarkable about this is the fact that Massachusetts Republicans are the most, or nearly the most, liberal Republicans in the entire country!

Very Jamesian, wouldn’t you say? And Boris’s was borne out by Scott Brown’s voting record, where he indeed was the most liberal of the Senate’s Republicans.

Political extrapolation

OK, now back to Sarah Palin. First, her popularity. Yes, Gov. Palin was popular, but Alaska is a small (in population) state, and surveys

find that most of the popular governors in the U.S. are in small states. Here are data from 2006 and 2008:

governors.png

There are a number of theories about this pattern; what’s relevant here is that a Bill James-style statistical adjustment might be necessary before taking state-level stats to the national level.

The difference between baseball and politics

There’s something else going on, though. It’s not just that Palin isn’t quite so popular as she appeared at first. There’s also a qualitative shift. From “boringly competent nonpartisan” to . . . well, leaving aside any questions of competence, she’s certainly no longer boring or nonpartisan! In baseball terms, this is like Ozzie Smith coming up from the minors and becoming a Dave Kingman-style slugger. (Please excuse my examples which reveal how long it’s been since I’ve followed baseball!)

So how does baseball differ from politics, in ways that are relevant to statistical forecasting?

1. In baseball there is only one goal: winning. Scoring more runs than the other team. Yes, individual players have other goals: staying healthy, getting paid, not getting traded to Montreal, etc., but overall the different goals are aligned, and playing well will get you all of these to some extent.

But there are two central goals in politics: winning and policy. You want to win elections, but the point of winning is to enact policies that you like. (Sure, there are political hacks who will sell out to the highest bidder, but even these political figures represent some interest groups with goals beyond simply being in office.)

Thus, in baseball we want to predict how a player can help his team win, but in politics we want to predict two things: electoral success and also policy positions.

2. Baseball is all about ability–natural athletic ability, intelligence (as Bill James said, that and speed are the only skills that are used in both offense and defense), and plain old hard work, focus, and concentration. The role of ability in politics is not so clear. In his remarks that started this discussion, Ubs suggested that Palin had the ability and inclination to solve real problems. But it’s not clear how to measure such abilities in a way that would allow any generalization to other political settings.

3. Baseball is the same environment at all levels. The base paths are the same length in the major leagues as in AA ball (at least, I assume that’s true!), the only difference is that in the majors they throw harder. OK, maybe the strike zone and the field dimensions vary, but pretty much it’s the same game.

In politics, though, I dunno. Some aspects of politics really do generalize. The Massachusetts Senate has got to be a lot different from the U.S. Senate, but, in their research, Boris Shor and Nolan McCarty have shown that there’s a lot of consistency in how people vote in these different settings. But I suspect things are a lot different for the executive, where your main task is not just to register positions on issues but to negotiate.

4. In baseball, you’re in or out. If you’re not playing (or coaching), you’re not really part of the story. Sportswriters can yell all they want but who cares. In contrast, politics is full of activists, candidates, and potential candidates. In this sense, the appropriate analogy is not that Sarah Palin started as Ozzie Smith and then became Dave Kingman, but rather a move from being Ozzie Smith to being a radio call-in host, in a world in which media personalities can be as powerful, and as well-paid, as players on the field. Perhaps this could’ve been a good move for, say, Bill Lee, in this alternative universe? A player who can’t quite keep the ball over the plate but is a good talker with a knack for controversy?

Commenter Paul made a good point here:

How many at-bats long is a governorship? The most granular I could imagine possibly talking is a quarter. At the term level we’d be doing better making each “at-bat” independent of the previous. 20 or so at-bats don’t have much predictive value either. Even over a full 500 at-bat season, fans try to figure out whether a big jump in BABIP is a sign of better bat control or luck.

The same issues arise at very low at-bat counts too. If you bat in front of a slugger, you can sit on pitches in the zone. If you’ve got a weakness against a certain pitching style, you might not happen to see it. And once the ball is in the air, luck is a huge factor in if it travels to a fielder or between them.

I suspect if we could somehow get a political candidate to hold 300-400 different political jobs in different states, with different party goals and support, we’d be able to do a good job predicting future job performance, even jumping from state to national levels. But the day to day successes of a governor are highly correlative.

Indeed, when it comes to policy positions, a politician has lots of “plate appearances,” that is, opportunities to vote in the legislature. But when it comes to elections, a politician will only have at most a couple dozen in his or her entire career.

All the above is from a post from 2011. I thought about it after this recent exchange with Mark Palko regarding the political candidacy of Ron DeSantis.

In addition to everything above, let me add one more difference between baseball and politics. In baseball, the situation is essentially fixed, and pretty much all that matters is player ability. In contrast, in politics, the most important factor is the situation. In general elections in the U.S., the candidate doesn’t matter that much. (Primaries are a different story.) In summary, to distinguish baseball players in ability we have lots of data to estimate a big signal; to distinguish politicians in vote-getting ability we have very little data to estimate a small signal.

On the border between credulity and postmodernism: The case of the UFO’s-as-space-aliens media insiders

I came across this post from Tyler Cowen:

From an email I [Cowen] sent to a well-known public intellectual:

I think the chance that the bodies turn out to be real aliens is quite low.

But the footage seems pretty convincing, a way for other people to see what…sources have been telling me for years. [Everyone needs to stop complaining that there are no photos!]

And to think it is a) the Chinese, b) USG secret project, or…whatever…*in Mexico* strains the imagination.

It is interesting of course how the media is not so keen to report on this. They don’t have to talk about the aliens, they could just run a story “The Mexican government has gone insane.” But they won’t do that, and so you should update your mental model of the media a bit in the “they are actually pretty conservative, in the literal sense of that term, and quite readily can act like a deer frozen in the headlights, though at some point they may lurch forward with something ill-conceived.”

Many of you readers are from Christian societies, or you are Christian. But please do not focus on the bodies! I know you are from your early upbringing “trained” to do so, even if you are a non-believer. Wait until that evidence is truly verified (and I suspect it will not be). Focus on the video footage.

In any case, the Mexican revelations [sic] mean this issue is not going away, and perhaps this will force the hand of the USG to say more than they otherwise would have.

The above-linked post seems ridiculous to me, while comments on the post are much more reasonable—I guess it’s not hard to be reasonable when all you have to do is laugh at a silly hoax.

From a straight-up econ point of view I guess it makes sense that there has been a continuing supply of purported evidence for space aliens: there’s a big demand for this sort of thing so people will create some supply. It’s disappointing to me to see someone as usually-savvy as Cowen falling for this sort of thing, but (a) there’s some selection bias, as I’m not writing about all the people out there who have not been snookered by this Bermuda triangle ancient astronauts Noah’s ark fairies haunted radios bigfoot ESP ghosts space aliens stuff.

Given my earlier post on news media insiders getting all excited about UFOs (also this), you won’t be surprised to hear that I’m annoyed by Cowen’s latest. It’s just so ridiculous! Amusingly, his phrasing, “I think the chance that the bodies turn out to be real aliens is quite low,” echoes that of fellow contrarian pundit Nate Silver, who wrote, “I’m not saying it’s aliens, it’s almost definitely not aliens.” Credit them for getting the probability on the right side of 50%, but . . . c’mon.

As I wrote in my earlier posts, what’s noteworthy is not that various prominent people think that UFO’s might be space aliens—as I never tire of saying in this context, 30% of Americans say they believe in ghosts, which have pretty much the same basis in reality—; rather, what’s interesting is that they feel so free to admit this belief. I attribute this to a sort of elite-media contagion: Ezra Klein and Tyler Cowen believe the space aliens thing is a possibility, they’re smart guys, so other journalists take it more seriously, etc. Those of us outside the bubble can just laugh, but someone like Nate Silver is too much of an insider and is subject to the gravitational pull of elite media, twitter, etc.

Mark Palko offers a slightly different take, attributing the latest burst of elite credulity to the aftereffects of a true believer who managed to place a few space-aliens-curious stories into the New York Times, which then gave the story an air of legitimacy etc.

The space aliens thing is interesting in part because it does not seem strongly connected to political polarization. You’ve got Cowen on the right, Klein on the left, and Silver on the center-left. OK, just three data points, but still. Meanwhile, Cowen gets a lot of far-right commenters, but most of the commenters to his recent post are with me on this one, just kind of baffled that he’s pushing the story.

Postmodernism

A couple days after seeing Cowen’s post, I happened to be reading a book that discussed postmodernism in the writing of history. I don’t care so much about postmodernism, but the book was interesting; I’ll discuss it in a future post.

In any case, here’s the connection I saw.

Postmodernism means different things to different people, but one of its key tenets is that there is no objective truth . . . uhhhh, let me just “do a wegman” here and quote wikipedia:

Postmodernism is an intellectual stance or mode of discourse which challenges worldviews associated with Enlightenment rationality dating back to the 17th century. Postmodernism is associated with relativism and a focus on the role of ideology in the maintenance of economic and political power. Postmodernists are “skeptical of explanations which claim to be valid for all groups, cultures, traditions, or races, and instead focuses on the relative truths of each person”. It considers “reality” to be a mental construct. Postmodernism rejects the possibility of unmediated reality or objectively-rational knowledge, asserting that all interpretations are contingent on the perspective from which they are made; claims to objective fact are dismissed as naive realism.

One thing that struck me about Cowen’s post was not just that he’s sympathetic to the space-aliens hypothesis; also it seems to bug him that the elite news media isn’t covering it more widely. Which is funny, because it bugs me that the media (including Bloomberg columnist Cowen) are taking it as seriously as they do!

Cowen writes, “It is interesting of course how the media is not so keen to report on this.” Doesn’t seem so interesting to me! My take is that most people in the media have some common sense and also have some sense of the history of this sort of nexus of hoaxes and credibility, from Arthur Conan Doyle onward.

The postmodernism that I see coming from Cowen is in statement, “the footage seems pretty convincing, a way for other people to see what . . . sources have been telling me for years,” which seems to me, as a traditional rationalist or non-postmodernist, to be a form of circular reasoning saying that something is real because people believe in it. Saying “this issue is not going away” . . . I mean, sure, astrology isn’t going away either! Unfortunately, just about nothing ever seems to go away.

Oppositionism

There’s something else going on here, it’s hard for me to put my finger on, exactly . . . something about belief in the occult as being oppositional, something “they” don’t want you do know about, whether “they” is “the media” or “the government” or “organized religion” or “the patriarchy” or “the medical establishment” or whatever. As we discussed in an earlier post on a topic, one interesting thing is how things happen that push certain fringe beliefs into a zone where it’s considered legitimate to take them seriously. As a student of public opinion and politics, I’m interested not just in who has these beliefs and why, but also in the processes by which some such beliefs but not others circulate so that they seem perfectly normal to various people such as Cowen, Silver, etc., in the elite news media bubble.

When Steve Bannon meets the Center for Open Science: Bad science and bad reporting combine to yield another ovulation/voting disaster

The Kangaroo with a feather effect

A couple of faithful correspondents pointed me to this recent article, “Fertility Fails to Predict Voter Preference for the 2020 Election: A Pre-Registered Replication of Navarrete et al. (2010).”

It’s similar to other studies of ovulation and voting that we’ve criticized in the past (see for example pages 638-640 of this paper.

A few years ago I ran across the following recommendation for replication:

One way to put a stop to all this uncertainty: preregistration of studies of all kinds. It won’t quell existing worries, but it will help to prevent new ones, and eventually the truth will out.

My reaction was that this was way too optimistic.The ovulation-and-voting study had large measurement error, high levels of variation, and any underlying effects were small. And all this is made even worse because they were studying within-person effects using a between-person design. So any statistically significant difference they find is likely to be in the wrong direction and is essentially certain to be a huge overestimate. That is, the design has a high Type S error rate and a high Type M error rate.

And, indeed, that’s what happened with the replication. It was a between-person comparison (that is, each person was surveyed at only one time point), there was no direct measurement of fertility, and this new study was powered to only be able to detect effects that were much larger than would be scientifically plausible.

The result: a pile of noise.

To the authors’ credit, their title leads right off with “Fertility Fails to Predict . . .” OK, not quite right, as they didn’t actually measure fertility, but at least they foregrounded their negative finding.

Bad Science

Is it fair for me to call this “bad science”? I think this description is fair. Let me emphasize that I’m not saying the authors of this study are bad people. Remember our principle that honesty and transparency are not enough. You can be of pure heart, but if you are studying a small and highly variable effect using a noisy design and crude measurement tools, you’re not going to learn anything useful. You might as well just be flipping coins or trying to find patterns in a table of random numbers. And that’s what’s going on here.

Indeed, this is one of the things that’s bothered me for years about preregistered replications. I love the idea of preregistration, and I love the idea of replication. These are useful tools for strengthening research that is potentially good research and for providing some perspective on questionable research that’s been done in the past. Even the mere prospect of preregistered replication can be a helpful conceptual tool when considering an existing literature or potential new studies.

But . . . if you take a hopelessly noisy design and preregister it, that doesn’t make it a good study. Put a pile of junk in a fancy suit and it’s still a pile of junk.

In some settings, I fear that “replication” is serving a shiny object to distract people from the central issues of measurement, and I think that’s what’s going on here. The authors of this study were working with some vague ideas of evolutionary psychology, and they seem to be working under the assumption that, if you’re interested in theory X, that the way to science is to gather some data that have some indirect connection to X and then compute some statistical analysis in order to make an up-or-down decision (“statistically significant / not significant” or “replicated / not replicated”).

Again, that’s not enuf! Science isn’t just about theory, data, analysis, and conclusions. It’s also about measurement. It’s quantitative. And some measurements and designs are just too noisy to be useful.

As we wrote a few years ago,

My criticism of the ovulation-and-voting study is ultimately quantitative. Their effect size is tiny and their measurement error is huge. My best analogy is that they are trying to use a bathroom scale to weigh a feather—and the feather is resting loosely in the pouch of a kangaroo that is vigorously jumping up and down.

At some point, a set of measurements is so noisy that biases in selection and interpretation overwhelm any signal and, indeed, nothing useful can be learned from them. I assume that the underlying effect size in this case is not zero—if we were to look carefully, we would find some differences in political attitude at different times of the month for women, also different days of the week for men and for women, and different hours of the day, and I expect all these differences would interact with everything—not just marital status but also age, education, political attitudes, number of children, size of tax bill, etc etc. There’s an endless number of small effects, positive and negative, bubbling around.

Bad Reporting

Bad science is compounded by bad reporting. Someone pointed me to a website called “The National Pulse,” which labels itself as “radically independent” but seems to be an organ of the Trump wing of the Republican party, and which featured this story, which they seem to have picked up from the notorious sensationalist site, The Daily Mail:

STUDY: Women More Likely to Vote Trump During Most Fertile Point of Menstrual Cycle.

A new scientific study indicates women are more likely to vote for former President Donald Trump during the most fertile period of their menstrual cycle. According to researchers from the New School for Social Research, led by psychologist Jessica L Engelbrecht, women, when at their most fertile, are drawn to the former President’s intelligence in comparison to his political opponents. The research occurred between July and August 2020, observing 549 women to identify changes in their political opinions over time. . . .

A significant correlation was noticed between women at their most fertile and expressing positive opinions towards former President Donald Trump. . . . the 2020 study indicated that women, while ovulating, were drawn to former President Trump because of his high degree of intelligence, not physical attractiveness. . . .

As I wrote above, I think that research study was bad, but, conditional on the bad design and measurement, its authors seem to have reported it honestly.

The news report adds new levels of distortion.

– The report states that the study observed women “to identify changes in their political opinions over time.” First, the study didn’t “observe” anyone; they conducted an online survey. Second, they didn’t identify any changes over time: the women in the study were surveyed only once!

– The report says something about “a significant correlation” and that “the study indicated that . . .” This surprised me, given that the paper itself was titled, “Fertility Fails to Predict Voter Preference for the 2020 Election.” How do you get from “fails to predict” to “a significant correlation”? I looked at the journal article and found the relevant bit:

Results of this analysis for all 14 matchups appear in Table 2. In contrast to the original study’s findings, only in the Trump-Obama matchup was there a significant relationship between conception risk and voting preference [r_pb (475) = −.106, p = .021] such that the probability of intending to vote for Donald J. Trump rose with conception risk.

Got it? They looked at 14 comparisons. Out of these, one of these was “statistically significant” at the 5% level. This is the kind of thing you’d expect to see from pure noise, or the mathematical equivalent, which is a study with noisy measurements of small and variable effects. The authors write, “however, it is possible that this is a Type I error, as it was the only significant result across the matchups we analyzed,” which I think is still too credulous a way to put it; a more accurate summary would be to say that the data are consistent with null effects, which is no surprise given the realistic possible sizes of any effects in this very underpowered study.

The authors of the journal article also write, “Several factors may account for the discrepancy between our [lack of replication of] the original results.” They go on for six paragraphs giving possible theories—but never once considering the possibility that the original studies and theirs were just too noisy to learn anything useful.

Look. I don’t mind a bit of storytelling: why not? Storytelling is fun, and it can be a good way to think about scientific hypotheses and their implications. The reason we do social science is because we’re interested in the social world; we’re not just number crunchers. So I don’t mind that the authors had several paragraphs with stories. The problem is not that they’re telling stories, it’s that they’re only telling stories. They don’t ever reflect that this entire literature is chasing patterns in noise.

And this lack of reflection about measurement and effect size is destroying them! They went to all this trouble to replicate this old study, without ever grappling with that study’s fundamental flaw (see kangaroo picture at the top of this post). Again, I’m not saying that they authors are bad people or that they intend to mislead; they’re just doing bad, 2010-2015-era psychological science. They don’t know better, and they haven’t been well served by the academic psychology establishment which has promoted and continues to promote this sort of junk science.

Don’t blame the authors of the bad study for the terrible distorted reporting

Finally, it’s not the authors’ fault that their study was misreported by the Daily Mail and that Steve Bannon associated website. “Fails to Predict” is right there in the title of the journal article. If clickbait websites and political propagandists want to pull out that p = 0.02 result from your 14 comparisons and spin a tale around it, you can’t really stop them.

The Center for Open Science!

Science reform buffs will enjoy these final bits from the published paper:

The New York Young Republican Club

This story hit the news yesterday:

Rep. Marjorie Taylor Greene (R-Georgia) entertained Republicans in Manhattan Saturday night with a range of one-liners trolling the political left on hot-button topics.

“I want to tell you something, if Steve Bannon and I had organized that, we would have won. Not to mention, we would’ve been armed,” she said of her role at the Jan. 6, 2021 attack on the US Capitol. . . .

The controversial congresswoman was one of several high-profile conservative firebrands . . . at the annual event hosted by the New York Young Republican Club.

Her speech took a strange turn while she noted how “you can pick up a butt plug or a dildo at Target nowadays” . . .

I was curious so I did a quick search:

OK, yeah, I guess she’s right!

Anyway, this all reminded me that I spoke at the Young Republican Club once! It was in February, 2009, and I talked about our book, Red State Blue State. It was a mellow occasion. As I recall they told me they were looking forward to an upcoming softball game with the Young Democrats. I remember telling the Young Republicans that I’d recently given a talk at the Princeton Club of New York, and that 50 years earlier there would’ve been a big overlap between the Princeton Club and the Young Republican Club but not anymore.

It seems that the organization has changed even more in the past decade or so. I can’t imagine speaking at a club where they joke about armed overthrow of the government. That really bothers me. I guess that’s how they were talking 55 years ago at the Young Communist Club, or the Students for a Democratic Society.

Stepping back, we can understand this as part of the residue of a couple hundred years of 1776 rhetoric. If it was ok for Sam Adams, George Washington, etc., to have an armed insurrection against the British, and if it was ok to have a bunch of slaveowners have an armed insurrection against the U.S. government in 1861, then what exactly is wrong with modern-day congressmembers talking about shooting up the Capitol building? Once you accept the idea that Joe Biden and Abraham Lincoln are worse than George III, the rest all follows. From that perspective, it makes me wonder why there isn’t more of this sort of talk in public. Ultimately I guess it’s more of a pragmatic issue than a moral issue. It’s against the law to threaten to shoot people, so keep talking like that and you might go to jail. Also most voters aren’t into the whole insurrection thing, so if you’re a politician and you’re not in a safe seat, this sort of extremism could be politically risky. But from a theoretical perspective, sure, if 1776 or 1861 is the standard, then, yeah, shooting at government officials could be considered to be just fine.

There’s a difference. King George was not elected by the public; Abraham Lincoln and Joe Biden were. But, once you accept the idea that overthrowing the government is OK, I guess it’s no big deal if guns are involved.

Back in 2009, nobody at the Young Republican Club was talking about hijacking Congress. Or butt plugs, for that matter. At least not on the day I was there. Things have changed.

Michael Lewis.

I just read this interesting review by Patrick Redford of the new book by journalist Michael Lewis on Sam Bankman-Fried, the notorious crypto fraudster.

We discussed earlier how the news media, including those such as Michael Lewis and Tyler Cowen who play the roles of skeptics in the media ecosystem, were not just reporters of the crypto fraud; they also played an important part in promoting and sustaining the bubble. As I wrote when all this came out, the infrastructure of elite journalism was, I think, crucial to keeping the bubble afloat. Sure, crypto had lots of potential just from rich guys selling to each other and throwing venture capital at it, and suckers watching Alex Jones or whatever investing their life savings, but elite media promotion took it to the next level.

We’ve talked earlier about the Chestertonian principle that extreme skepticism is a form of credulity, an idea that seems particularly relevant to the comedian and political commentator Joe Rogan, whose twin stances of deep skepticism and deep credulity are inextricably intertwined. To be skeptical about the moon landing or the 2020 election requires belief in all sorts of ridiculous theories and discredited evidence. Skepticism and credulity here are not opposites—we’re not talking “horseshoe theory”—; rather, they’re the same thing. Skepticism of the accepted official view that the moon landings actually happened, or that the laws of physics are correct and ghosts don’t exist, or that UFOs are not space aliens, or that Joe Biden won the 2020 election by 7 million votes, is intimately tied to active belief in some wacky theory or unsubstantiated or refuted empirical claim.

I’m not saying that skepticism is always a form of credulity, just that sometimes it is. When I was skeptical of the Freakonomics-endorsed claim that beautiful parents are 36% more likely to have girls, no credulity was required, just some background in sex-ratio statistics and some basic understanding of statistics. Similarly if you want to be skeptical of the claim that UFOs are space aliens etc. There’s ordinary skepticism and credulous skepticism. Ordinary skepticism, though, it easy to come by. Credulous skepticism, by its nature, is a more unstable quantity and requires continuing effort—you have to carefully protect your skeptical beliefs and keep them away from any stray bits of truth that might contaminate them. Which I guess is one reason that people such as Rogan who have the ability to do this with a straight face are so well compensated.

But what about Michael Lewis? Like everybody else, I’m a fan of Moneyball. I haven’t read any of his other books—I guess a lot of his books are about rich financial guys, and, while I know the topic is important, it’s never interested me so much—but then last year he interviewed me for a podcast! I was kinda scared at first—my previous experiences with journalists reporting on scientific controversies have been mixed, and I didn’t want to be walking into a trap—but it worked out just fine. Lewis was straightforward, with no hidden agenda. The podcast worked out just fine. Here’s a link to the podcast, and here’s an article with some background. Perhaps I should’ve been more suspicious given that the podcast is produced by a company founded by plagiarism-defender

Of Lewis’s new book, Redford writes:

A common thematic thread, perhaps the common thread, wending throughout Michael Lewis’s bibliography is the limits of conventional wisdom. His oeuvre is stuffed with stories about the moments at which certain bedrock ideas—ones about finance, or baseball, or electoral politics—crumble under their own contradictions. This is helped along, often, by visionary seers—like Michael Burry, Billy Beane, or John McCain—who put themselves in position to take advantage of those who are either too blinkered or too afraid to see the unfamiliar future taking shape in front of them.

That describes Moneyball pretty accurately, and at first it would seem to fit a podcast called “Against the Rules,” but actually that podcast was all about how there was existing expertise, settled enough to be considered “conventional wisdom,” that was swept aside in a wave of confusion. In particular, Lewis talked about the Stanford covid crew, a group of well-connected iconoclasts in the Billy Beane mode, but he showered them with criticism, not praise. Maybe that podcast worked because he was going against type? I don’t know.

Just speaking in general terms, we shouldn’t ignore the visionary seers—Bill James, imperfect though he may have been, was really on to something, and even his missteps are often interesting. But we can’t assume the off-the-beaten-path thinkers are always right: that way lies Freakonomics-style madness, as here and here.

It’s too bad what happened to Lewis with the Bankman-Fried thing, but I wouldn’t attribute it to a general problem with his take on conventional wisdom. It’s more of an omnipresent risk of journalism, which is to frame a story around a hero, which creates problems if the hero isn’t even a plausible anti-hero. (Recall that an “anti-hero” is not the opposite of a hero; rather, he’s someone who doesn’t look or act like a conventional hero but still is a hero in some sense.)

Stabbers gonna stab — fraud edition

One of the themes of Dan Davies’s book, Lying for Money, was that fraudsters typically do their crimes over and over again, until they get caught. And then, when they are released from prison, they do it again. This related to something I noticed in the Theranos story, which was that the fraud was in open sight for many years and the fraudsters continued to operate in the open.

Also regarding that interesting overlap of science and business fraud, I noted:

There seem to have been two ingredients that allowed Theranos to work. And neither of these ingredients involved technology or medicine. No, the two things were:

1. Control of the narrative.

2. Powerful friends.

Neither of these came for free. Theranos’s leaders had to work hard, for long hours, for years and years, to maintain control of the story and to attract and maintain powerful friends. And they needed to be willing to lie.

The newest story

Ben Mathis-Lilley writes:

On Wednesday, the Department of Justice announced that it has arrested a 48-year-old Lakewood, New Jersey, man named Eliyahu “Eli” Weinstein on charges of operating, quote, “a Ponzi scheme.” . . . How did authorities know that Weinstein was operating a Ponzi scheme? For one thing, he allegedly told associates, while being secretly recorded, that he had “Ponzied” the money they were using to repay investors. . . . Weinstein is further said to have admitted while being recorded that he had hidden assets from federal prosecutors. (“I hid money,” he is said to have told his conspirators, warning them that they would “go to jail” if anyone else found out.) . . .

These stories of “least competent criminals” are always fun, especially when the crime is nonviolent so you don’t have to think too hard about the victims.

What brings this one to the next level is the extreme repeat-offender nature of the criminal:

There was also one particular element of Weinstein’s background that may have alerted the DOJ that he was someone to keep an eye on—namely, that he had just been released from prison after serving eight years of a 24-year sentence for operating Ponzi schemes. More specifically, Weinstein was sentenced to prison for operating a Ponzi scheme involving pretend real estate transactions, then given a subsequent additional sentence for operating a second Ponzi scheme, involving pretend Facebook stock purchases, that he conducted after being released from custody while awaiting trial on the original charges.

Kinda like when a speeding driver runs over some kid and then it turns out the driver had 842 speeding tickets and the cops had never taken away his car, except in this case there’s no dead kid and the perp had already received a 24-year prison sentence.

How is it that he got out after serving only 8 years, anyway?

In January 2021, Weinstein was granted clemency by President Donald Trump at the recommendation of, among others, “the lawyer Alan Dershowitz,” who has frequently been the subject of news coverage in recent years for his work representing Trump and his relationship with the late Jeffrey Epstein.

Ahhhhh.

This all connects to my items #1 and 2 above.

The way Weinstein succeeded (to the extent he could be considered a success) at fraud was control of the narrative. And he got his get-out-of-jail-free card from his powerful friends. “Finding your roots,” indeed.

Stabbers gonna stab

This all reminded me of a story that came out in the newspaper a few decades ago. Jack Henry Abbott was a convicted killer who published a book while in prison. Abbott’s book was supposed to be very good, and he was subsequently released on parole with the support of various literary celebrities including Norman Mailer. Shortly after his release, Abbott murdered someone else and returned to prison, where he spent the rest of his life.

The whole story was very sad, but what made it particularly bizarre was that Abbott’s first murder was a stabbing, his second murder was a stabbing, and his most prominent supporter, Mailer, was notorious for . . . stabbing someone.

Lefty Driesell and Bobby Knight

This obit of the legendary Maryland basketball coach reminded me of a discussion we had a few years ago. It started with a remark in a published article by political scientist Diana Mutz identifying herself as “a Hoosier by birth and upbringing, the daughter of a former Republican officeholder, and someone who still owns a home in Mike Pence’s hometown.”

That’s interesting: I don’t know so many children of political officeholders! Actually, I can’t think of anyone I know, other than Mutz, who is a child of a political officeholder, but perhaps there are some such people in my social network. I don’t know the occupations of most of my friends’ parents.

Anyway, following up on that bit from Mutz, sociologist Steve Morgan added some background of his own:

I was also born in Indiana, and in fact my best friend in the 1st grade, before I left the state, was Pat Knight. To me, his father, Bobby Knight was a pleasant and generally kind man (who used to give us candy bars, etc.). He turned out to be a Trump supporter, and probably his son too. So, in addition to not appreciating his full basketball personality when I was 6 years old, I also did not see his potential to find a demagogue inspiring. We moved to Ohio, where I received a lot of education in swing-state politics and Midwestern resentment of coastal elites.

And then I threw in my two cents:

I was not born in Indiana, but I grew up in suburban Maryland (about 10 miles from Brett Kavanaugh, but I went to a public school in a different part of the county and so had zero social overlap with his group). One of the kids in my school was Chuck Driesell, son of Lefty Driesell, former basketball coach at the University of Maryland. Lefty is unfortunately now most famous for his association with Len Bias, but Chuck and I were in high school before that all happened, when Lefty was famous for being a good coach who couldn’t ever quite beat North Carolina. Once I remember the Terps decided to beat Dean Smith at his own game by doing the four corners offense themselves. But it didn’t work; I think Maryland ended up losing 21-18 or some other ping-pong-like score. Chuck was in my economics class. I have no idea if he’s now a Trump supporter. I guess it’s possible. One of the other kids in that econ class was an outspoken conservative, one of the few Reagan supporters of our group of friends back in 1980. Chuck grew up and became a basketball coach; the other kid grew up and became an economist.

I never went to a Maryland basketball game all the time I lived there, even when I was a student at the university. I wish I’d gone; I bet it would’ve been a lot of fun. My friends and I played some pickup soccer and basketball, and I watched lots of sports on TV, but for whatever reason we never even considered the idea of going to a game. We didn’t attend any of high school football games either, even though our school’s team was the state champions. This was not out of any matter of principle; we just never thought of going. Our loss.

The Lakatos soccer training

Alex Lax writes:

While searching the Internet for references to Lakatos, I noticed your comment about Lakatos being a Stalinist. I met Imre Lakatos shortly after his arrival in the UK. My parents spoke Hungarian and helped to settle the refugees to 1956. Imre Lakatos was one of those the refugees. I remember him playing football with me at a time when Hungarian football was seen as far superior to English football, and I also remember once when we met him at Cambridge railway station with his latest girlfriend who was very tall. She had managed to lose some contact lenses and I was grovelling around on the road trying to find them. During his visits he would often complain about his treatment in prison which destroyed his stomach and he would rant against the Communists. However after his death, I was told that a book by a well known French Communist was dedicated to Imre. I have not found this dedication but if true would suggest that he was a Communist of some flavour while pretending otherwise.

I hope this might be of interest to you.

He adds:

By the way, the Lakatos soccer training consisted of two players on a small pitch with two smallish opposing goals, with each player protecting their own goal. Each player was only allowed to touch the ball once.

I’m interested in Lakatos because his writing has been very influential to my work; see for example here and here. He was said to be a very difficult person, but perhaps that was connected in some way to his uncompromising intellectual nature, which served him well as an innovator in the philosophy of science.

If school funding doesn’t really matter, why do people want their kid’s school to be well funded?

A question came up about the effects of school funding and student performance, and we were referred to this review article from a few years ago by Larry Hedges, Terri Pigott, Joshua Polanin, Ann Marie Ryan, Charles Tocci, and Ryan Williams:

One question posed continually over the past century of education research is to what extent school resources affect student outcomes. From the turn of the century to the present, a diverse set of actors, including politicians, physicians, and researchers from a number of disciplines, have studied whether and how money that is provided for schools translates into increased student achievement. The authors discuss the historical origins of the question of whether school resources relate to student achievement, and report the results of a meta-analysis of studies examining that relationship. They find that policymakers, researchers, and other stakeholders have addressed this question using diverse strategies. The way the question is asked, and the methods used to answer it, is shaped by history, as well by the scholarly, social, and political concerns of any given time. The diversity of methods has resulted in a body of literature too diverse and too inconsistent to yield reliable inferences through meta-analysis. The authors suggest that a collaborative approach addressing the question from a variety of disciplinary and practice perspectives may lead to more effective interventions to meet the needs of all students.

I haven’t followed this literature carefully. It was my vague impression that studies have found effects of schools on students’ test scores to be small. So, not clear that improving schools will do very much. On the other hand, everyone wants their kid to go to a good school. Just for example, all the people who go around saying that school funding doesn’t matter, they don’t ask to reduce the funding of their own kids’ schools. And I teach at an expensive school myself. So lots of pieces here, hard for me to put together.

I asked education statistics expert Beth Tipton what she thought, and she wrote:

I think the effect of money depends upon the educational context. For example, in higher education at selective universities, the selection process itself is what ensures success of students – the school matters far less. But in K-12, and particularly in under resourced areas, schools and finances can matter a lot – thus the focus on charter schools in urban locales.

I guess the problem here is that I’m acting like the typical uninformed consumer of research. The world is complicated, and any literature will be a mess, full of claims and counter-claims, but here I am expecting there to be a simple coherent story that I can summarize in a short sentence (“Schools matter” or “Schools don’t matter” or, maybe, “Schools matter but only a little”).

Given how frustrated I get when others come into a topic with this attitude, I guess it’s good for me to recognize when I do it.

Social penumbras predict political attitudes (my talk at Harvard on Monday Feb 12 at noon)

Monday, February 12, 2024, 12:00pm to 1:15pm

Social penumbras predict political attitudes

The political influence of a group is typically explained in terms of its size, geographic concentration, or the wealth and power of the group’s members. This article introduces another dimension, the penumbra, defined as the set of individuals in the population who are personally familiar with someone in that group. Distinct from the concept of an individual’s social network, penumbra refers to the circle of close contacts and acquaintances of a given social group. Using original panel data, the article provides a systematic study of various groups’ penumbras, focusing on politically relevant characteristics of the penumbras (e.g., size, geographic concentration, sociodemographics). Furthermore, we show the connection between changes in penumbra membership and public attitudes on policies related to the group.

This is based on a paper with Yotam Margalit from 2021.

“Replicability & Generalisability”: Applying a discount factor to cost-effectiveness estimates.

This one’s important.

Matt Lerner points us to this report by Rosie Bettle, Replicability & Generalisability: A Guide to CEA discounts.

“CEA” is cost-effectiveness analysis, and by “discounts” they mean what we’ve called the Edlin factor—“discount” is a better name than factor, because it’s a number that should be between 0 and 1, it’s what you should multiply a point estimate by to adjust for inevitable upward biases in reported effect-size estimates, issues discussed here and here, for example.

It’s pleasant to see some of my ideas being used for a practical purpose. I would just add that type M and type S errors should be lower for Bayesian inferences than for raw inferences that have not been partially pooled toward a reasonable prior model.

Also, regarding empirical estimation of adjustment factors, I recommend looking at the work of Erik van Zwet et al; here are some links:
What’s a good default prior for regression coefficients? A default Edlin factor of 1/2?
How large is the underlying coefficient? An application of the Edlin factor to that claim that “Cash Aid to Poor Mothers Increases Brain Activity in Babies”
The Shrinkage Trilogy: How to be Bayesian when analyzing simple experiments
Erik van Zwet explains the Shrinkage Trilogy
The significance filter, the winner’s curse and the need to shrink
Bayesians moving from defense to offense: “I really think it’s kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?”
Explaining that line, “Bayesians moving from defense to offense”

I’m excited about the application of these ideas to policy analysis.

A new argument for estimating the probability that your vote will be decisive

Toby Ord writes:

I think you will like this short proof that puts a lower bound on the probability that one’s vote is decisive.

It requires just one assumption (that the probability distribution over vote share is unimodal) and takes two inputs (the number of voters & the probability the underdog wins). It shows that in (single level) elections that aren’t forgone conclusions, the chance your vote is decisive can’t be much lower than 1 in the number of voters (and I show where some models that say otherwise go wrong).

Among other things, this makes it quite plausible that the moral value of voting is positive in expectation, since the aggregate value scales with n, while the probability scales with 1/n. Voting would produce net-value roughly when the value of your preferred candidate to the average citizen exceeds the cost to you of voting.

This relates to my paper with Edlin and Kaplan, “Voting as a rational choice: why and how people vote to improve the well-being of others.”

I was happy to see that Ord’s article mentioned the point made in the appendix of our 2004 paper, as it addresses a question that often arises, which is whether a vote can never be decisive because when an election is close there can be a recount.

Also, some background: it’s my impression that the p = 10^-90 crowd (that is, the people who assign ridiculously small probabilities of a single vote being decisive) are typically not big fans of the idea of democracy, so it is convenient for them to suppose that voting doesn’t matter.

I’m not saying the p = 10^-90 people are cynical, as they may sincerely believe that democracy is overrated, and then this is compounded by innumeracy. Probability is difficult!

And then there’s just the general issue that people seem to have the expectation that, when there’s any sort of debate, that all the arguments must necessarily go in their favor, so they’ll twist and turn a million ways to avoid grappling with contrary arguments; see for example this discussion thread where I tried to clarify a point but it didn’t work.

Regarding this last point, Ord writes:

I hope it is useful to have a simple formula for a completely safe lower bound for the chance a vote is decisive. Not the same as your empirically grounded versions, but nice to show people who don’t trust the data or the more complex statistical analysis.

Hey, I got tagged by RetractoBot!

A message came in my inbox from “The RetractoBot Team, University of Oxford,” with subject line, “RetractoBot: You cited a retracted paper”:

That’s funny! When we cited that paper by Lacour and Green, we already knew it was no good. Indeed, that’s why we cited it. Here’s the relevant paragraph from our article:

In political science, the term “replication” has traditionally been applied to the simple act of reproducing a published result using the identical data and code as used in the original analysis. Anyone who works with real data will realize that this exercise is valuable and can catch problems with sloppy data analysis (e.g., the Excel error of Reinhart and Rogoff 2010, or the “gremlins” article of Tol 2009, which required nearly as many corrections as the number of points in its dataset; see Gelman 2014). Reexamination of raw data can also expose mistakes, such as the survey data of LaCour and Green (2014); see Gelman (2015).

We also cited two other notorious papers, Reinhart and Rogoff (2010) and Tol (2009), both of which should have been retracted but are still out there in the literature. According to Google scholar, Reinhard and Rogoff (2010) has been cited more than 5000 times! I guess that many of these citations are from articles such as mine, using it as an example of poor workflow, but still. Meanwhile, Tol (2009) has been cited over 1500 times. It does have a “correction and update” from 2014, but that hardly covers its many errors and inconsistencies.

Anyway, I can’t blame RetractoBot for not noticing the sense of my citation; it’s just funny how they sent that message.

Sympathy for the Nudgelords: Vermeule endorsing stupid and dangerous election-fraud claims and Levitt promoting climate change denial are like cool dudes in the 60s wearing Che T-shirts and thinking Chairman Mao was cool—we think they’re playing with fire, they think they’re cute contrarians pointing out contradictions in the system. For a certain kind of person, it’s fun to be a rogue.

A few months ago I wrote about some disturbing stuff I’d been hearing about from Harvard Law School professors Cass Sunstein and Adrian Vermeuele. The two of them wrote an article back in 2005 writing, “a refusal to impose [the death] penalty condemns numerous innocent people to death. . . . a serious commitment to the sanctity of human life may well compel, rather than forbid that form of punishment. . . .”

My own view is that the death penalty makes sense in some settings and not others. To say that “a serious commitment to the sanctity of human life may well compel” the death penalty . . . jeez, I dunno, that’s some real Inquisition-level thinking going on. Not just supporting capital punishment, they’re compelling it. That’s a real edgelord attitude, kinda like the thought-provoking professor in your freshman ethics class who argues that companies have not just the right but the moral responsibility to pollute the maximum amount possible under the law because otherwise they’re ducking their fiduciary responsibility to the shareholders. Indeed, it’s arguably immoral to not pollute beyond the limits of the law if the expected gain from polluting is lower than the expected loss from getting caught and fined.

Sunstein and Vermeule also recommended that the government should fight conspiracy theories by engaging in “cognitive infiltration of extremist groups,” which seemed pretty rich, considering that Vermeule spent his online leisure hours after the 2020 election promoting election conspiracy theories. Talk about the fox guarding the henhouse. This is one guy I would not trust to be in charge of government efforts to cognitively infiltrate extremist groups!

Meanwhile, these guys go on NPR, they’ve held appointive positions with the U.S. government, they’re buddies with elite legal academics . . . it bothers me! I’m not saying their free speech should be suppressed—we got some Marxists running around in this country too—I just don’t want them anywhere near the levers of power.

Anyway, I heard by email from someone who knows Sunstein and Vermeuele. It seems that both of them are nice guys, and when they stick to legal work and stay away from social science or politics they’re excellent scholars. My correspondent also wrote:

And on that 2019 Stasi tweet. Yes, it was totally out of line. You and others were right to denounce it. But I think it’s worth pointing out that he deleted the tweet the very same day (less than nine hours later), apologized for it as an ill-conceived attempt at humor, and noted with regret that the tweet came across as “unkind and harsh to good people doing good and important work.” I might gently and respectfully suggest that continuing to bring up this tweet four years later, after such a prompt retraction—which was coupled with an acknowledgement of the value of the work that you and others are doing in focusing on the need for scrutiny and replication of eye-catching findings—might be perceived as just a tad ungracious, even by those who believe that Cass was entirely in the wrong and you were entirely in the right as regards the original tweet. To paraphrase one of the great capital defense lawyers (who obviously said this in a much more serious context), all of us are better than our worst moment.

I replied:

– Regarding the disjunction between Vermeule’s scholarly competence and nice-guyness, on one hand, and his extreme political views: I can offer a statistical or population perspective. Think of a Venn diagram where the two circles are “reasonable person” and “extreme political views and actions.” (I’m adding “actions” here to recognize that the issue is not just that Vermeule thinks that a fascist takeover would be cool, but that he’s willing to sell out his intellectual integrity for it, in the sense of endorsing ridiculous claims.)

From an ethical point of view, there’s an argument in favor of selling out one’s intellectual integrity for political goals. One can make this argument for Vermeule or also for, say, Ted Cruz. The argument is that the larger goal (a fascist government in the U.S., or more power for Ted Cruz) is important enough that it’s worth making such a sacrifice. Or, to take slightly lesser examples, the argument would be that when Hillary Clinton lied about her plane being shot at, or when Donald Trump lied about . . . ok, just about everything, that they were thinking about larger goals. Indeed, one could argue that for Cruz and the other politicians, it’s not such a big deal—nobody expects politicians to believe half of what they’re saying anyway—but for Vermeule to trash his reputation in this way, that shows real commitment!

Actually, I’m guessing that Vermeule was just spending too much time online in a political bubble, and he didn’t really think that endorsing these stupid voter-fraud claims meant anything. To put it another way, you and I think that endorsing unsubstantiated claims of voting fraud is bad for two reasons: (1) intellectually it’s dishonest to claim evidence for X when you have no evidence for X, (2) this sort of thing is dangerous in the short term by supplying support to traitors, and (3) it’s dangerous in the long term by degrading the democratic process. But, for Vermeule, #2 and #3 might well be a plus not a minus, and, as for #1, I think it’s not uncommon for people to make a division between their professional and non-professional statements, and to have a higher standard for the former than the latter. Vermeule might well think, “Hey, that’s just twitter, it’s not real.” Similarly, the economist Steven Levitt and his colleagues wrote all sorts of stupid things (along with many smart things) under the Freakonomics banner, thinks which I guess (or, should I say, hope) he’d never have done in his capacity as an academic. Just to be clear, I’m not saying that everyone does this, indeed I don’t think I do it—I stand by what I blog, just as I stand by my articles and books—but I don’t think everyone does. Another example that’s kinda famous is biologists who don’t believe in evolution. They can just separate the different parts of their belief systems.

Anyway, back to the Venn diagram. The point is that something like 30% of Americans believe this election fraud crap. 30% of Americans won’t translate into 30% of competent and nice-guy law professors, but it won’t be zero, either. Even if it’s only 10% or less in that Venn overlap, it won’t be zero. And the people inside that overlap will get attention. And some of them like the attention! So at that point you can get people going further and further off the deep end.

If it would help, you could think of this as a 2-dimensional scatterplot rather than a Venn diagram, and in this case you can picture the points drifting off to the extreme over time.

To look at this another way, consider various well-respected people in U.S. and Britain who were communists in the 1930s through 1950s. Some of these people were scientists! And they said lots of stupid things. From a political perspective, that’s all understandable: even if they didn’t personally want to tear up families, murder political opponents, start wars, etc., they could make the case that Stalin’s USSR was a counterweight to fascism elsewhere. But from an intellectual perspective, they wouldn’t always make that sort of minimalist case. Some of them were real Soviet cheerleaders. Again, who knows what moral calculations they were making in their heads.

I’m not gonna go all Sunstein-level contrarian and argue that selling out one’s intellectual integrity is the ultimate moral sacrifice—I’m picturing a cartoon where Vermeule is Abraham, his reputation is Isaac, and the Lord is thundering above, booming down at him to just do it already—but I guess the case could be made, indeed maybe will be the subject of one of the 8 books that Sunstein comes out with next year and is respectfully reviewed on NPR etc.

– Regarding the capital punishment article: I have three problems here. The first is their uncritical acceptance of a pretty dramatic claim. In Sunstein and Vermeule’s defense, though, back in 2005 it was standard in social science for people to think that statistical significance + identification strategy + SSRN or NBER = discovery. Indeed, I’d guess that most academic economists still think that way! So to chide them on their innumeracy here would be a bit . . . anachronistic, I guess. The second problem is that I’m guessing they were so eager to accept this finding is that it allowed them to make this cool point that they wanted to make. If they’d said, “Here’s a claim, maybe it’s iffy but if it’s true, it has some interesting ethical implications…”, that would be one thing. But that’s not what I read their paper as saying. By saying “Recent evidence suggests that capital punishment may have a significant deterrent effect” and not considering the opposite, they’re making the fallacy of the one-way bet. My third problem is that I think their argument is crap, even setting aside the statistical study. I discussed this a bit in my post. There are two big issues they’re ignoring. The first is that if each execution saves 18 lives, then maybe we should start executing innocent people! Or, hey, we can find some guilty people to execute, maybe some second-degree murderers, armed robbers, arsonists, tax evaders, speeders, jaywalkers, . . . . shouldn’t be too hard to find some more targets–after all, they used to have the death penalty for forgery. Just execute a few hundred of them and consider how many lives will be saved. That may sound silly to you, but it’s Sunstein and Vermeule, not me, who wrote that bit about “a serious commitment to the sanctity of human life.” I discussed the challenges here in more detail in a 2006 post; see the section, “The death penalty as a decision-analysis problem?” My point is not that they have to agree with me, just that it’s not a good sign that their long-ass law article with its thundering about “the sanctity of human life” is more shallow than two paragraphs of a blog post.

In summary regarding the death-penalty article, I’m not slamming them for falling for crappy research (that’s what social scientists and journalists did back in 2005, and lots of them still to do this day) and I’m not slamming them for supporting death penalty (I’ve supported it too, at various times in my life; more generally I think it depends on the situation and that the death penalty can be a good idea in some circumstances, even if the current version in the U.S. doesn’t work so well). I’m slamming them for taking half-assed reasoning and presenting it as sophisticated. I’d say they don’t know better, they’re just kinda dumb—but you assure me that Vermeule is actually smart. So my take on it is that they’re really good at playing the academic game. For me to criticize their too-clever-by-half “law and economics” article as not being well thought through, that would be like criticizing LeBron James for not being a golf champion. They do what’s demanded of them in their job.

– Regarding Sunstein’s ability to learn from error: Yes, I mention in my post that Sunstein was persuaded by the article by Wolfers and Donohue. I do think it was good that Sunstein retracted his earlier stance. That’s one reason I was particularly disappointed by what he and his collaborator did in the second edition of Nudge, which was to memory-hole the Wansink episode. It was such a great opportunity in the revision, for them to have said that the nudge idea is so compelling that they (and many others) were fooled, and to consider the implications: in a world where people are rewarded for discovering apparently successful nudges, the Wansinks of the world will prosper, at least in the short term. Indeed, Sunstein and Thaler could’ve even put a positive spin on it by talking about the self-correcting nature of science, sunlight is the best disinfectant, etc. But, no, instead they remove it entirely, and then Sunstein returns to his previous credulous self by posting something on what he called the “coolest behavioral finding of 2019.” Earlier they’d referred to Wansink as having had multiple masterpieces. Kind of makes you question their judgment, no? My take on this is . . . for them, everyone’s a friend, so why rock the boat? As I wrote, it looks to me like an alliance of celebrities. I’m guessing that they are genuinely baffled by people like Uri Simonsohn or me who criticize this stuff: Don’t we have anything better to do? It’s natural to think of behavior of Simonsohn, me, and other “data thugs” as being kinda pathological: we are jealous, or haters, or glory-seekers, or we just have some compulsion to be mean (the kind of people who, in another life, would be Stasi).

– Regarding the Stasi quote: Yes, I agree it’s a good thing Sunstein retracted it. I was not thrilled that in the retraction he said he’d thought it had “a grain of truth,” but, yeah, as retractions go, it was much better than average! Much better than the person who called people “terrorists,” never retracted or apologized, then later published an article lying about a couple of us (a very annoying episode to me, which I have to kind of keep quiet about cos nobody likes a complainer, but grrrr it burns me up, that people can just lie in public like that and get away with it). So, yes, for sure, next time I write about this I will emphasize that he retracted the Stasi line.

– Libertarian paternalism: There’s too much on this for one email, but for my basic take, see this post, in particular the section “Several problems with science reporting, all in one place.” This captures it: Sunstein is all too willing to think that ordinary people are wrong, while trusting the testimony of Wansink, who appears to have been a serial fabricator. It’s part of a world in which normies are stupidly going about their lives doing stupid things, and thank goodness (or, maybe I should say in deference to Vermeule, thank God) there are leaders like Sunstein, Vermeule, and Wansink around to save us from ourselves, and also in the meantime go on NPR, pat each other on the back on Twitter, and enlist the U.S. government in their worthy schemes.

– People are complicated: Vermeule and Sunstein are not “good guys” or “bad guys”; they’re just people. People are complicated. What makes me sad about Sunstein is that, as you said, he does care about evidence, he can learn from error. But then he chooses not to. He chooses to stay in his celebrity comfort zone, making stupid arguments evaluating the president’s job performance based on the stock market, cheerleading biased studies about nudges as if they represent reality. See the last three paragraphs here. Another bad thing Sunstein did recently was to coauthor that Noise book. Another alliance of celebrities! (As a side note, I’m sad to see the collection of academic all-star endorsements that this book received.) Regarding Sunstein himself, see the section “A new continent?” of that post. As I wrote at the time, if you’re going to explore a new continent, it can help to have a local guide who can show you the territory.

Vermeule I know less about; my take is that he’s playing the politics game. He thinks that on balance the Republicans are better than the Democrats, and I’m guessing that when he promotes election fraud misinformation, that he just thinks he’s being mischievous and cute. After all, the Democrats promoted misinformation about police shootings or whatever, so why can’t he have his fun? And, in any case, election security is important, right? Etc etc etc. Anyone with a bit of debate-team experience can justify lots worse than Vermeule’s post-election tweets. I guess they’re not extreme enough for Sunstein to want to stop working with him.

– Other work by Vermeule and Sunstein: They’re well-respected academics, also you and others say how smart they are, so I can well believe they’ve also done high-quality work. It might be that their success in some subfields led them into a false belief that they know what they’re doing in other areas (such as psychology research, statistics, and election administration) where they have no expertise. As the saying goes, sometimes it’s important to know what you don’t know.

My larger concern, perhaps, is that these people get such deference in academia and the news media, that they start to believe their own hype and they think they’re experts in everything.

– Conspiracy theories: Sunstein and Vermeule wrote, “Many millions of people hold conspiracy theories; they believe that powerful people have worked together in order to withhold the truth about some important practice or some terrible event. A recent example is the belief, widespread in some parts of the world, that the attacks of 9/11 were carried out not by Al Qaeda, but by Israel or the United States.” My point here is that there are two conspiracy theories here: a false conspiracy theory that the attacks were carried out by Israel or the United States, and a true conspiracy theory that the attacks were carried out by Al Qaeda. In the meantime, Vermeule has lent his support to unsupported conspiracy theories regarding the 2020 election. So Vermeule is incoherent. On one hand, he’s saying that conspiracy theories are a bad thing. On the other hand, in one place he’s not recognizing the existence of true conspiracies; in another place he’s supporting ridiculous and dangerous conspiracy theories, I assume on the basis that they are in support of his political allies. I don’t think it’s a cheap shot to point out this incoherence.

And what the does it mean that Sunstein thinks that “Because those who hold conspiracy theories typically suffer from a ‘crippled epistemology,’ in accordance with which it is rational to hold such theories, the best response consists in cognitive infiltration of extremist groups.”—but he continues to work with Vermeule? Who would want to collaborate with someone who suffers from a crippled epistemology (whatever that means)? The whole thing is hard for me to interpret except as an elitist position where some people such as Sunstein and Vermeule are allowed to believe whatever they want, and hold government positions, while other people get “cognitively infiltrated.”

– The proposed government program: I see your point that when the government is infiltrating dangerous extremist groups, it could make sense for them to try to talk some of these people out of their extremism. After all, for reasons of public safety the FBI and local police are already doing lots of infiltration anyway—they hardly needed Sunstein and Vermeule’s encouragement. Overall I suspect it’s a good thing that the cops are gathering intelligence this way rather than just letting these groups make plans in secret, set off bombs, etc., and once the agents are on the inside, I’d rather have them counsel moderation than do that entrapment thing where they try to talk people into planning crimes so as to be able to get more arrests.

I think what bothers me about the Sunstein and Vermeule article—beyond that they’re worried about conspiracy theories while themselves promoting various con artists and manipulators—is in their assumption that the government is on the side of the good. Perhaps this is related to Sunstein being pals with Kissinger. I labeled Sunstein and Vermeuele as libertarian paternalists, but maybe Vermuele is better described as an authoritarian; in any case they seem to have the presumption that the government is on their side, whether it’s for nudging people to do good things (not to do bad things) or for defusing conspiracy theories (not to support conspiracy theories).

But governments can’t always be trusted. When I wrote, “They don’t even seem to consider a third option, which is the government actively promoting conspiracy theories,” it’s not that I was saying that this third option was a good thing! Rather, I was saying that the third option is something that’s actually done, and I gave examples of the U.S. executive branch and much of Congress in the period Nov 2020 – Jan 2021, and the Russian government in their invasion of Ukraine.” And it seems that Vermeule may well be cool with both these things! So my reaction to Vermeule saying the government should be engaging in information warfare is similar to my reaction when the government proposed to start a terrorism-futures program and have it be run by an actual terrorist: it might be a good idea in theory and even in practice, but (a) these are not the guys I would want in charge of such a program, and (b) their enthusiasm for it makes me suspicious.

– Unrelated to all the above: You say of Vermeule, “after his conversion to Catholicism, he adopted the Church’s line on moral opposition to capital punishment.” That’s funny because I thought the Catholic church was cool with the death penalty—they did the inquisition, right?? Don’t tell me they’ve flip-flopped! Once they start giving into the liberals on the death-penalty issue, all hell will break loose.

OK, why did I write all that?

1. The mix of social science, statistical evidence, and politics is interesting and important.

2. As an academic, I’m always interested in academics behaving badly, especially when it involves statistics or social science in some way. In particular, the idea that these guys are supposed to be so smart and so nice in regular life, and then they go with these not-so-smart, not-so-nice theories, that’s interesting. When mean, dumb people promote mean, dumb ideas, that’s not so interesting. But when nice, smart people do it . . .

3. It’s been unfair to Sunstein for me to keep bringing up that Stasi thing.

Regarding item 2, one analogy I can see with Vermeule endorsing stupid and dangerous election-fraud claims is dudes in the 60s wearing Che T-shirts and thinking Chairman Mao was cool. From one perspective, Che was one screwed-up dude and Mao was one of history’s greatest monsters . . . but both of them were bad-ass dudes and it was cool to give the finger to the Man. Similarly, Vermeule could well think of Trump as badass, and he probably thinks its hilarious to endorse B.S. claims that support his politics. Kinda like how Steven Levitt probably thinks he’s a charming mischievous imp by supporting climate denialists. Levitt would not personally want his (hypothetical) beach house on Fiji to be flooded, but, for a certain kind of person, it’s fun to be a rogue.

Here’s what I wrote when the topic came up before:

There’s no evidence that Vermeule was trying to overthrow the election. He was merely supportive of these efforts, not doing it himself, in the same way that an academic Marxist might root for the general strike and the soviet takeover of government but not be doing anything active on the revolution’s behalf.