Someone pointed me to a recent post by Nate Silver, “Polling averages shouldn’t be political litmus tests, and they need consistent standards, not make-it-up-as-you-go,” where Nate wrote:
The new Editorial Director of Data Analytics at ABC News, G. Elliott Morris, who was brought in to work with the remaining FiveThirtyEight team, sent a letter to the polling firm Rasmussen Reports demanding that they answer a series of questions about their political views and polling methodology or be banned from FiveThirtyEight’s polling averages, election forecasts and news coverage. I found several things about the letter to be misguided. . . .
First, I strongly oppose subjecting pollsters to an ideological or political litmus test. . . . Why, unless you’re a dyed-in-the-wool left-leaning partisan, would having a “relationship with several right-leaning blogs and online media outlets” lead one to “doubt the ethical operation of the polling firm”? . . .
Rasmussen has indeed had strongly Republican-leaning results relative to the consensus for many years. Despite that strong Republican house effect, however, they’ve had roughly average accuracy overall because polls have considerably understated Republican performance in several recent elections (2014, 2016, 2020). . . . Is that a case of two wrongs making a right — Rasmussen has had a Republican bias, but other polls have had a Democratic bias, so they come out of the wash looking OK? Yeah, probably. Still, there are ways to adjust for that — statistical ways like a house effects adjustment . . .
Second, even if you’re going to remove Rasmussen from the averages going forward, it’s inappropriate to write them out of the past . . . It’s bad practice to revise data that’s already been published, based on decisions you made long after that data was published. For one thing, it makes your numbers less reliable as a historical record. For another, it can lead to overconfidence when using that data to train or build models. . . .
Third, I think it’s clear that the letter is an ad hoc exercise to exclude Rasmussen, not an effort to develop a consistent set of standards. . . . The thing about running a polling average is that you need a consistent and legible set of rules that be applied to hundreds of pollsters you’ll encounter over the course of an election campaign. Going on a case-by-case basis is a) extremely time-consuming . . . and b) highly likely to result in introducing your own biases . . . Perhaps Morris’s questions were getting at some larger theme or more acute problem. But if so, he have should stated it more explicitly in his letter. . . .
Nate raises several interesting questions here:
1. Is there any good reason for a relationship with “right-leaning” outlets such as Fox News and Steve Bannon to cause one to “doubt the ethical operation of the polling firm”?
2. Does it ever make sense to remove a biased poll, rather than including in your analysis with a statistical correction?
3. If you are changing your procedure going forward, is it a mistake to make those changes retroactively on past work?
4. Is it appropriate to send a letter to one polling organization without going through the equivalent process with all the other pollsters whose data you’re using?
Any followups?
I’ll go through the above questions one at a time, but first I was curious if Nate or Elliott had said anything more on the topic.
I found these two items on twitter:
– This from Elliott: “asking pollsters detailed methodological questions is not (or shouldn’t be!) controversial. it’s standard practice in most media organizations, and aggregators should probably even be publishing responses for the public and using them as a way to gauge potential measurement error,” linking to a list of questions that CNN asks of all pollsters.
– This from Nate, referring to Elliott’s letter to Rasmussen as a “Spanish Inquisition” and linking to this article from the Washington Examiner which, among other things, reported this from a Rasmussen poll:
Whaaaaa? As a check, I googled *abortion roe wade polling* and found some recent items:
Gallup: “As you may know, the Supreme Court overturned its 1973 Roe versus Wade decision concerning abortion, meaning there is no Constitutional protection for abortion rights and each state could set its own laws to allow, restrict or ban abortions. Do you think overturning Roe versus Wade was a good thing or a bad thing?”: 38% “good thing,” 61% “bad thing,” 1% no opinion.
CBS/YouGov: “Last year, the U.S. Supreme Court ended the constitutional right to abortion by overturning Roe v. Wade. Do you approve or disapprove of the Court overturning Roe v. Wade?”: 44% “approve,” 56% “disapprove.”
USA Today (details here): “It’s been a year since the Supreme Court overturned the Roe v. Wade decision, eliminating a
constitutional right to an abortion at some stages of pregnancy. Do you support or oppose the court decision to overturn Roe v. Wade?”: 30% “support,” 58% “oppose,” 12% undecided.
There’s other polling out there, all pretty much consistent with the above. An then there’s Rasmussen, which stands out. Would I want to include Rasmussen’s “Majority Now Approve SCOTUS Abortion Ruling” in a polling average? I’m not sure.
Some of it must could be their question wording: “Last year, the Supreme Court overturned the 1973 Roe v. Wade decision, so that each state can now determine its own laws regarding abortion. Do you approve or disapprove of the court overturning Roe v. Wade?” This isn’t far from the Gallup question, but they does remove the “Constitutional protection” phrase, and I guess that could make a difference. Also, they’re just counting “likely voters,” and much could depend on where those respondents come from.
Whether or not it makes sense to take the Rasmussen organization seriously (I remain concerned about their numbers that added up to 108%), I think it’s kinda journalistic malpractice for the Washington Examiner to report their claim of “Support for overturning Roe v. Wade is up since last year. 52% to 44%, US likely voters approve,” without even noting how much that disagrees with all other polling out there. My first thought was that, yeah, the Washington Examiner is a partisan outlet, but even partisans benefit from accurate news, right? I guess the point is that the role of an operation such as the Washington Examiner is not so much to inform readers as to circulate talking points and get them out into the general discussion—indeed, thanks to Nate and then me, it happened here!
1. Is there any good reason for a relationship with “right-leaning” outlets such as Fox News and Steve Bannon to cause one to “doubt the ethical operation of the polling firm”?
OK, now on to Nate’s questions. First, should we doubt the ethics of a pollster who hangs out with Fox News and Steve Bannon? My answer here is . . . it depends!
On one hand, . . . Should we discredit my statistical work because I teach at Columbia University, an institution whose most famous professor was Dr. Oz and which notoriously promulgated false statistics for its college rankings? Lots of people teach at Columbia, similarly lots of people go on Fox News: there’s an appeal to reaching an audience of millions. Going on Fox might be a bad idea, but does it cast doubt on a pollster’s ethics?
As I said, it depends. If a pollster or quantitative social scientist is consistently using crap statistics to push election denial, then, yes, I do doubt their ethics. The relevant point here is not that Fox and Bannon are “right-leaning” but rather that they’ve been fueling election denial misinformation, and distorted election statistics are part of the process.
So, yeah, I agree with Nate that Elliott’s phrase, “several right-leaning blogs and online media outlets,” doesn’t tell the whole story—as Nate put it, “Perhaps Morris’s questions were getting at some larger theme or more acute problem.” There is a larger theme and more acute problem, and that’s refuted claims about the election that have been endorsed by major political and media figures. Given what Rasmussen’s been doing in this area, I think Nate’s been a bit too quick to take their side of the story on this, to refer to Elliott’s inquiries as an “inquisition,” etc. You don’t have to be a “dyed-in-the-wool left-leaning partisan” to doubt the ethical operation of a polling firm that is promoting lies about the election.
How close does a pollster need to be to election deniers so that I don’t trust it at all? I don’t know. I guess it depends on context, which is a good reason for Elliott to ask specific questions to Rasmussen about their polling methodology. If they’re open about what they’re doing, that’s a good sign; if they give no details, that’s gonna make it harder to trust them. Rasmussen has no duty to respond to those questions, Fivethirtyeight has no duty to include its polls in their analyses, etc etc all down the line.
2. Does it ever make sense to remove a biased poll, rather than including in your analysis with a statistical correction?
Discarding a data point is equivalent to including it but giving it a weight of zero or, from a Bayesian point of view, allowing it to be biased with an infinite-variance prior on the bias. So we can transform Nate’s very reasonable implied question (why discard Rasmussen polls? Why not just include your skepticism in your model?) as the question: Why not just give the Rasmussen polls a very small weight or, from a Bayesian point of view, allow them to have a bias that has a very large uncertainty?
There are two answers here. The first is that if the weight is very small or the bias has a huge uncertainty, then it’s pretty much equivalent to not including the survey at all. Remember 13. The second answer is that if these surveys are really being manipulated, then there’s no reason to think the bias is consistent. To put it another way: if you don’t think the Rasmussen polls are providing useful information, then you might not want to include them for the same reason that you wouldn’t include a rotten onion in your stew. Sure, one bad onion won’t destroy the taste—it’ll be diluted amid all the other flavors (including those of all the non-rotten onions you’ve thrown in)—but what’s the point?
This second answer is as much procedural as substantive: by excluding a pollster entirely, Fivethirtyeight is saying they don’t want to be using numbers that they can’t, on some level, trust. They’re making the procedural point that they have some rules for the polls they include, some red lines that cannot be crossed.
From the other direction, Nate’s plea for Fivethirtyeight to continue including Rasmussen’s polls in its analyses is also a procedural and perception-based argument: he’s making the procedural point that “you need a consistent and legible set of rules” and can’t be making case-by-case decisions.
The funny thing is . . . Nate and Elliott are kind of saying the same thing! Elliott’s saying they’ll be removing Rasmussen unless they follow the rules and Nate’s saying that too. I looked up Fivethirtyeight’s rules for pollsters from when Nate was running the organization and it says “Pollsters must also be able to answer basic questions about their methodology, including but not limited to the polling medium used (e.g., landline calls, text, etc.), the source of their voter files, their weighting criteria, and the source of the poll’s funding.” And they don’t include “‘Nonscientific’ polls that don’t attempt to survey a representative sample of the population or electorate.” So I guess a lot depends on the details; see item 4 below.
3. If you are changing your procedure going forward, is it a mistake to make those changes retroactively on past work?
I have a lot of sympathy for Nate’s argument here. He created the Fivethirtyeight polling averages, then combined this with his interest in sports analytics, worked his butt off for over a decade . . . and now the new team is talking about changing things. It would be kind of like if CRC Press hired someone to create a fourth edition of Bayesian Data Analysis, and the new author decided to remove chapter 6 because it didn’t match his philosophy. I’d be furious! OK, that’s not a perfect analogy because my coauthors and I have copyright on BDA, but the point is that Nate was Fivethirtyeight for awhile, so it’s frustrating to think of the historical record being changed.
That said, it’s not clear to me that Elliott is planning to change the historical record. From his quoted letter: “If banned, Rasmussen Reports would also be removed from our historical averages of polls and from our pollster ratings. Your surveys would no longer appear in reporting and we would write an article explaining our reasons for the ban.” It could be that the polls would still be in the database, just flagged and not included in the averages. I think that would be OK.
To put it another way, I think it’s ok to go back and clean up old data, as long as you’re transparent about it.
From a slightly different angle, Nate writes, “There’s also an implicit conflict here about the degree to which journalists should gatekeep or shield the public from potential sources of ‘misinformation.'” I’m not exactly sure of Elliott’s motivations here, but my guess is that his goal is not so much to “shield the public” but rather to come up with more accurate forecasts. Nate argues that including a Republican-biased poll should lead to more accurate forecasts by balancing other polls with systematic polling errors favoring the Democrats. I guess that if Fivethirtyeight going forward is not going to include Rasmussen polls, they’ll have to adjust for possible systematic errors in some other way. That would make sense to me, actually. If you do want to adjust for the possibility of errors on the scale of 2016 or 2020 (polls that showed the Democrats getting approximately 2.5 percentage points more support than they actually received in the vote), then it would make sense to make that adjustment straight up, without relying on Rasmussen to do it for you.
4. Is it appropriate to send a letter to one polling organization without going through the equivalent process with all the other pollsters whose data you’re using?
I have no idea what’s been going on between Fivethirtyeight and Rasmussen and between Fivethirtyeight and other polling organizations. The quoted letter from Elliott to Rasmussen begins, “I am emailing you to send a final notice . . .”, so it seems safe to assume this is just one in a series of communications, and we haven’t seen the others that came before.
Nate writes, “I think it’s clear that the letter is an ad hoc exercise to exclude Rasmussen, not an effort to develop a consistent set of standards.” My guess is that it’s neither an ad hoc exercise to exclude Rasmussen, nor an effort to develop a consistent set of standards, but rather that it’s an effort to apply an imperfect set of standards. Rules such as “Pollsters must also be able to answer basic questions about their methodology, including but not limited to . . .” and “‘Nonscientific’ polls that don’t attempt to survey a representative sample” are imperfect—but that’s the nature of rules.
I guess what I’m saying is that it’s hard to compare Fivethirtyeight’s interactions with Rasmussen with their interactions with other pollsters, given that (a) we don’t know what their interactions with Rasmussen are, and (b) we don’t what their interactions with other pollsters are.
Let me just say that this sort of thing is always challenging, as there’s no way to have completely consistent rules. For example, we have good reasons to be suspicious that Brian Wansink ever used his famous bottomless soup bowl in any actual experiment. Do we apply this level of scrutiny to the apparatus described in every peer-reviewed research article? No, first because this would require an immense amount of effort, and second because “this level of scrutiny” is not even defined. It’s judgment calls all the way down. Fivethirtyeight has a necessarily ambiguous policy on what polls they will include in their analyses—there’s no way for such a policy to not have some ambiguity—and Nate and Elliott are making different judgment calls on whether Rasmussen violates the policy.
Having this discussion
Unfortunately there hasn’t been much of a conversation on this poll-inclusion issue, which I guess is no surprise given that Nate (indirectly) called Elliott a bullshitter and explicitly writes, “I don’t intend this a back-and-forth.” Which is too bad, given that we’ve had good conversations on forecasting before.
It’s easier for me to have this discussion because I know both Nate and Elliott. I don’t know either of them well on a personal level, but I’ve collaborated with both of them (for example, here and here) and I think they both do great work. I’ve criticized Nate’s forecasting procedure; then again, I’ve also criticized Elliott’s, even though (or especially because) it was done in collaboration with me.
To say I like both of them is not an attempt to put myself above the fray or to characterize their disagreements as minor. People often get themselves into positions where they are legitimately angry at each other—it’s happened to me plenty of times! The main point of the present post is that the decisions Elliott is making regarding which polls to include in his analysis, and the questions that Nate is asking, are challenging, with no easy answers.
P.S. Here’s a brief summary of statistical concerns with the 2020 presidential election forecasts from Economist and Fivethirtyeight forecasts. tl;dr: both had problems, in different ways.
All of those Roe v Wade poll questions are politically biased.
The thing that got overturned was the right to medical privacy. The point was that whatever medical treatments you have undergone (or plan to undergo) is none of the government’s business.
Now the door has been opened for all sorts of banning and mandating interventions.
“Should we discredit my statistical work because I teach at Columbia University, an institution whose most famous professor was Dr. Oz and which notoriously promulgated false statistics for its college rankings?”
There’s a whole spectrum of possibilities between “take uncritically” and “discredit.” Keeping that in mind, yes, I would treat your work more skeptically because of Columbia’s failings in that regard. Universities have a gatekeeper function with their faculty — supposedly — where they represent that they’ve hired honest scholars. Evidence that they’ve failed in that function casts doubt on everyone there.
Total:
I think the appropriate lesson to take regarding the story of Dr. Oz at Columbia is not that Columbia failed in its “gatekeeper function.” Columbia has no gatekeeper function and does not attempt to check that the people who works here are honest. Rather, the lesson to take is that, upon hearing of unethical behavior by one of its faculty, Columbia might be very slow to do anything about it.
Putting all this together: No, I don’t think that Dr. Oz being at Columbia should cause you to be more skeptical about statements coming from researchers at Columbia. However, if a Columbia professor is credibly accused of research misconduct, you shouldn’t necessarily take the fact that the professor hasn’t been fired as a sign that nothing bad was going on.
Of course Columbia is a gatekeeper. You check credentials of job candidates, right? Ask for references? You don’t hire just anyone, on the assumption that the things you require translated into honesty (among other things)
On the second part, could you tell me how to distinguish between a Columbia professor who has been accused of dishonesty but Columbia has hushed things up, and an honest one? Is there a hand sign?
Total:
Let me separate this into two parts.
First, Dr. Oz. Columbia hired him to do surgery, and at the time of his hiring I think that’s what he did. The promotion of pseudoscience came later. No amount of gatekeeping or vetting or whatever would’ve caught the problem at the time of his hiring, as there was no problem them. Later, he became famous, first as a benign expert and then for the pseudoscience. Then there was a problem. The difficulty is that Columbia couldn’t easily discipline or fire him for this.
More generally, the problem usually seems to be things the faculty member does after being hired. There are some counterexamples such as Ariely having problems at MIT and then being hired by Duke, but mostly it seems to be existing junior or senior faculty doing suspicious things while under employment, in which case there is no relevant gatekeeping function. Again, the problem is more that it can be difficult to get rid of problematic faculty.
Let me put this back into one part. Columbia hired people who then committed egregious fraud. Columbia’s reputation is built in part on the belief that it is a community of scholars dedicated to truth. That turns out not to be true.
You and your reputation is affected by that as well. Your house is not in order and you don’t get to handwave that away.
Its also good to know that tenure-track hiring decisions at universities are almost totally controlled by faculty in the relevant department. Other parts of the bureaucracy decide when they get allocated funds to hire someone, but who they hire is almost entirely in the hands of the hiring committee from the department. And even before tenure, people outside that academic field have almost no say on their promotion or continued employment. So its worthwhile to judge the credibility of academic fields, but judging the credibility of universities is not very helpful beyond a quick “are they part of a global academic community which polices itself, or just a university-shaped object?”
Sean:
Yes, I agree. It’s a bit weird to distrust my research because I work at an institution that did not fire Dr. Oz back in 2015 or whatever. It would make more sense to distrust my research because I work at an institution that faked its college ranking data for many years, all the way up to when the problem was publicly revealed in 2022. Or because I got my undergraduate degree in a department that was later notorious for continuing to employ a professor who worked in the generally-disbelieved area of cold fusion. Or because that institution was later notorious for selling its reputation to a notorious fraudster and sex offender. Or because I got my graduate degree at an institution that was later notorious for promoting the fraudulent “Jesus’s wife” manuscript, even after it was debunked by outside experts. Or because I worked for several years in a department that employed several professors who later became notorious for sexual harassment. Or because I’m a member of a statistical association that refused to retract a prestigious award after it was revealed that it had been given to a notorious plagiarist. Or because I’ve published in a journal that refused to correct or retract a paper with blatant falsehoods. Or at the very least you could distrust the various things I’ve published in a journal that’s notorious for publishing junk science. Or that I’ve worked for the military of a country that’s committed notorious war crimes in my lifetime. Or that I’m the citizen of a country whose former president seems to have committed major crimes. Or that I’m a member of a species that’s notorious for fraud in business, science, and other domains.
Or, closer to home, distrust my research because I’ve had to correct nearly 1% of my published papers.
Given all that, I highly recommend that you assess my research for plausibility. Don’t believe that it’s correct just because it’s published. Our houses are not in order and that can’t be handwaved away.
“Its also good to know that tenure-track hiring decisions at universities are almost totally controlled by faculty in the relevant department.”
You know, one of the least impressive rhetorical dodges TT faculty have managed over the past several decades of adjunctification and the corporatization of higher ed is the “It’s not me, it’s the administration/Board/etc. I have no control over *that.*”
“It’s a bit weird to distrust my research because I work at an institution that did not fire Dr. Oz back in 2015 or whatever. It would make more sense to distrust my research because I work at an institution that faked its college ranking data for many years, all the way up to when the problem was publicly revealed in 2022.”
Oh, I do that, too.
You keep wanting to separate *your* reputation from the reputation of the institution you work for, but I’m here telling you that that’s not possible. First, part of your reputation *comes* from the institution you work for — the vast majority of people have no idea who you are but a “Columbia University professor.” Something that harms Columbia’s reputation, harms yours. Second, when an example like Oz shows up, the question immediately pops for all CU professors: do they have similar issues? Were there concerns about them that were hushed up? Probably not in a major way, but it has an effect.
In short, no matter how much you want to pick and choose what parts of Columbia your identity is connected to, the larger public is not necessarily going to cooperate.
I’ve always found institutions unimpressive and am constantly amazed at the whole “Columbia Professor” bullshit. Show me some well thought out papers or books or even blog posts and my assessment goes up about infinitely more than knowing someone is from Harvard or whatever. I believe I’m in a tiny minority though. Argument from Authority appears to be almost the entirety of some peoples strategy for figuring out the world.
Total:
What Daniel said. Beyond that, you say that I keep wanting to separate my reputation from the reputation of the institution I work for . . . But, no, I’m not trying to do that! If I was bothered enough that Columbia wouldn’t fire Dr. Oz, I would’ve quit. Similarly, if I was bothered enough by various U.S. government policies, I would’ve stopped getting government grants. If I was bothered enough by the American Statistical Association not retracting the award they gave to the plagiarist, I’d quit the organization. Etc. My willingness to continue working at Columbia, to continue to take U.S. government funding, etc., represents some acceptance with these institutions. It’s right there on my webpage that I work at Columbia. Not only that, the url of this blog is “stat.columbia.edu”! Draw from that whatever conclusions you will.
Interesting discussion!!
However, I think everyone is missing the point. There’s no reason to trust Columbia or its researchers any less because of Oz or the ranking scandal. The brutal reality is that controversies like this are now par for the course in US academia. Institutions – that is, their leadership and administrations – lie and cheat. Faculty lie, cheat and fabricate research. Institutions and their leadership ignore all of this unless the press or someone else shoves their face in the pile, then they respond with undisguised reluctance.
The appropriate response – which is happening already – is to put less trust academia and academics as whole.
chipmunk
Fine – I agree we should trust academics less. But the problem is: who should we put our trust in? The reaction of many appears to be to trust worse sources than academics. Sort of like democracy being a terrible system until you look at the alternatives.
Dale:
AFAIK, originally the point of experiment and science was to *show* what is true, and thus dispense with trust and belief, because trust and belief – as we’re finding out – can be very unreliable. So I guess my perspective is that less trust in academia is a good thing. Belief and trust are shaky ground compared to science and reason.
But don’t confuse science and reason with *scientists* and academics – scientists and academics are people, and people are prone to failure, bias, and dishonesty. So when you say “the reaction of many appears to be to trust worse sources than academics,” you seem to be overlooking the harm that comes from trusting academics.
There’s one especially obvious example – a fraudulent and universal academic refrain that is causing so much stress *people are seeking mental health care because of it*. Right? Do I need to identify it?
There are plenty of others. Can you name one successful education initiative of the past 70 years? Almost ever major academic-backed education program has failed. For all the resources expended and all the certainty, a big fat zero. Can you name one successful “affirmative action” initiative? Here we are, sixty years on from the onset of affirmative action, and the groups that were supposed to have benefited from it are at best barely better off than before. Need I point out that the most ardent promoters of affirmative action are universities??? Is it not obvious how this is harmful? And don’t even get me started on CRT. It makes me laugh when academics mock creationists for claiming evolution is “just a theory”, then they go trotting out Critical Race “Theory” – as though it’s anything but a transparently bad ad-hoc claim – so easy to refute it doesn’t even rise to the level of a bad hypothesis.
So don’t go claiming that academics are an unequivocal source of “better” information. They do their fair share of damage. Skepticism of academics is not only justifiable, it’s necessary for a better society.
I hope you enjoyed your romp through your favorite complaints about the people and views you don’t agree with. The connection to trusting academics or trusting other sources is lost on me. I’m not saying it is good to trust academics. But I am saying people often ending up trusting worse sources. Nothing you have said in your tirade contributes to that discussion at all.
The real advice should be to trust nobody unless you have investigated it yourself. However, the world is too complicated for that to work for most people (perhaps all people). So we are stuck with having to trust others. I think you and I can agree that such trust should not be based on: credentials, fame, money, political views, or appearance (yes?). That might be all we agree on, however. Comparing the damage done by CRT vs damage done by Fox News might not produce the same agreement.
Tl;dr: I see a connection between Andrew (AG) and Dr Oz (MO), but the connection is negligibly weak.
I like both Total’s and Andrew’s arguments. While I am persuaded by Andrew’s argument (Columbia is not a gatekeeper) because of the academic freedom policy, I am somewhat reluctant to let Columbia off the hook without an argument: After all, the university itself (represented by committees and representatives) decides who to hire and/or grant tenure to, and can easily distance itself publicly from the statements of its employees. Therefore, I see a certain responsibility on the part of the universities when their professors make outlandish claims and are not held accountable. It reflects on all of Columbia’s employees, and even on the entire profession of the researcher in question. However, I see four mitigating factors that largely exonerate most of the large number of people I accused in the last sentence:
1. Researchers who say wrong or nonsensical things do not necessarily start their careers that way. They may have been sensible or even brilliant in the past. Removing a tenured professor is hard by design, as a safeguard against muzzling opinions critical of the government.
2. The larger the organisation, the greater the likelihood of ‘rotten apples’. As the hiring manager of a bank explained to me on the day its headquarters were raided by the police: ‘We have more than a hundred thousand employees. Of course we employ some criminals!’
3. The larger the organisation, the greater the expected (for lack of a better word) distance between any employees. At my university, there is no significant operational overlap between the researchers in the ‘Probability and Statistics Group’ of the Mathematics Department and the ‘Institute for Statistics and Econometrics’, or between the latter and the ‘Department of Agricultural Economics’. There is some overlap in their research, but they hardly ever work together. How far then is the distance from one researcher to another from a completely different field of research!
4. On a personal (not institutional) level, everyone is responsible for their own actions. A famous biblical saying mocks the concept of blaming the people around: ‘The fathers have eaten sour grapes, and the children’s teeth are set on edge.’
For these reasons, I have no problem with Andrew’s credibility, even on a theoretical level. There is obviously some connection between AG and MO, but it is extremely weak. I had never made the connection before. Now that I do, it does not weaken my belief in Andrew’s academic integrity and ability in the least. I guess we all agree at least on that last conclusion:)
Rasmussen just did a poll in March, 2023
https://www.rasmussenreports.com/public_content/politics/public_surveys/most_arizona_voters_believe_election_irregularities_affected_outcome
and finds that
“Of the 92% of Arizona voters who say they voted in the 2022 election, the new survey found 51% voted for Lake and 43% voted for Hobbs, while five percent (5%) say they voted for some other candidate.”
“This survey of actual Arizona voters, with a 3% margin of error, indicates that 8% more of them voted for Lake than voted for Katie Hobbs,” said Richard Thomas, National Chairman of Republicans United.
In other words, because the (Rasmussen) poll of today is correct, the election which actually took place has been, yet again, proven fraudulent.
Paul:
I assume these people don’t really believe these claims and that they’re just trying to keep “the base” happy. On the other hand, there must be some overlap between the credulous masses and the elite manipulators.
My best guess (I have no evidence here, it’s just my opinion) is that it’s somewhat in between, that people like this don’t have a strong take on the numbers being true or false. They might well think that “everybody does it,” that all polls are biased and that if they fiddle with the numbers to get the answers they want, it’s just part of the process.
Perhaps it’s kinda like how I described scientific misconduct:
So when people such as the National Chairman of Republicans United make this sort of ridiculous claim, perhaps it’s some version, from their point of view, of “fighting fire with fire.” Again, I don’t really know, this is just speculation. As a statistician, it just interests me when people are so sloppy with data in a public way. “Stupid” and “lying” are not the only options, but maybe most of the available options like on a convex combination of the two.
Andrew:
As always, you are being too kind when you say, “these people don’t really believe these claims and that they’re just trying to keep “the base” happy.” Working back from the desired outcome, 2023 Republican victory in Arizona (that is what this after-the-fact poll says), it makes perfect sense that this election was stolen. After all, that is the go-to solution du jour. Where the “jour” is every time and every place.
In their minds, the weather forecast is perfectly accurate and it is the weather which is wrong.
Correction
> In their minds, the weather forecast is perfectly accurate and it is the weather which is wrong.
The forecast (from Trump) was perfect but the weather was rigged by woke libz. Lest you foregot:
https://tinyurl.com/mr4bdvz4
Interestingly enough, fivethirtyeight (presumably Nate Silver, at the time, so I’ll say Silver from here out) did just ban a different pollster earlier this year – https://fivethirtyeight.com/features/why-were-preemptively-banning-a-pollster-and-not-banning-another/ . As far as the four points here applied to that case:
1. Silver decided to ban the director of the polling firm but not the firm itself. It sounds like they made the distinction because the firm was behaving ethically (to Silver’s eyes) in terms of divorcing itself from the director and following up. One wonders what they would have done if the pollster hadn’t acted the same way. In Rasmussen’s case, the firm itself seems to have issues from the link you provided.
2 and 3. Silver decided to ban future polls by the director but they don’t say anything about removing previous polls. Presumably that means they left them in. They seemed satisfied that the director didn’t influence the polls, so I guess they must be on the up and up. It’s also unclear what exactly fivethirtyeight’s investigation entailed; they might just be taking the pollster’s word for things. But again, one wonders what Silver would have done if the answer was less clear-cut. If the pollster was less forthcoming, or couldn’t say one way or the other if the director had influenced previous polls, would Silver still keep them in their averages? He didn’t have to make a decision.
4. From the article, they only reached out to this pollster after someone else reported on the director’s behavior. Essentially, something came to their attention about this one pollster and so they conducted an investigation… on this one pollster. I would guess they didn’t send all the other pollsters in their database a letter, let alone conduct an investigation, asking if their directors were behaving ethically or not. I would say that Silver is backwards re:Rasmussen here.
Alex:
Interesting story. Now I’m thinking that both Elliott and Nate have made a mistake by using the phrase “ban” as if Fivethirtyeight is some authority figure with the power to ban. Maybe it would be better for them just to say “will not use.” So, instead of “We are banning your poll,” they can say, “We will not be using your poll.” Then it’s much more clear that they’re making a decision about what data to use in their analysis, nothing more than that!
Yep. Agree here. Talk of “banning” is unnecessarily incendiary.
FiveThirtyEight is not just a forecaster, and I think in this case that explains the language. The FiveThirtyEight team rates polls. They also consume polls, both for forecasting and for journalistic work. For a forecaster or data journalist, it is indeed wrong to talk about a ban, because they have no authority. As a poll rating agency, they are indeed in a position to ban pollsters from their service. I usually see FiveThirtyEight as forecasters and journalists, but apparently the authors of the article in question were speaking from the perspective of a pollster rating agency (first sentence).
*(first sentence)= first sentence of the FiveThirtyEight article: ‘FiveThirtyEight is in the business of grading and reporting on polls — so when a pollster allegedly engages in misconduct, we take it seriously.’
That’s true. They’re just ‘banning’ polls from 538. There are obviously a ton of other poll consumers out there. And readers probably understand that implicitly, but it could be nice to say it more explicitly.
Dunno what’s so odd about Rasmussen poll results adding up to 108%. There were four choices, and if the results for each choice had a +/- 3% margin of error, the sum could have been as large as 112% and still be consistent:)
I originally considered a similar question of how much data to use in terms of tree rings used for paleoclimate, but election polls are very much the same.
If we imagine the final vote count (or paleoclimate proxy data) to be objective truth, then we could order each poll (or tree) in terms of fidelity to reality. I’ll stick with trees now. We can imagine that one of the trees we want to measure for climate response has the highest fidelity to past climate, and if we knew the fidelity, we could rank all the other trees below the best one in that order.
So if we know which tree is best, how much value would be added by adding additional trees to get an average? Here we have a sort of fuzzy notion that if we add another tree, it might smooth out some deviations from reality in the data of the first tree. So where the best tree zigs, the second-best zags, and we get a better composite than we originally had. But as we add more trees going down our list, we also know that each additional set of measurements has lower overall fidelity to reality than the previous ones. If we are continuously comparing our composite to reality as we add trees, we are hoping the fidelity of the average will rise for a while but we also know it will eventually start to get worse. At what point do we stop adding trees to our composite… and why? Now what happens if we don’t know which trees have the highest fidelity to begin with?
As the title of this post says, not easy. Certainly, if you have reason to believe that one pollster has their thumb on the scale and others do not, we can safely assume that that particular poll is well down our list in terms of fidelity, and not use it. Objective quality measures are all we have to solve the dilemma. The idea that errors will cancel out at the top of the list was a bit dubious to begin with, hoping that they will still be cancelling out when we are adding more from the bottom of the list is wishful thinking.
Your comment makes me wonder about the relationship between random forests, Bayesian Model Averaging, and this post. The amazing thing about random forests is that they work best if all the potential data is used regardless of its individual accuracy – the power comes from the averaging of independent models (more precisely, it is the averaging of the same model type, but with random choices of predictors). Bayesian Model Averaging suggests weighting models based on there measured accuracy. Tree rings and political polls seem to raise questions about which models to “count” and which to leave out, based on some measure of accuracy. This leave me wondering what the relationship is between these 3 approaches.
Dale wrote:
“The amazing thing about random forests is that they work best if all the potential data is used regardless of its individual accuracy – the power comes from the averaging of independent models (more precisely, it is the averaging of the same model type, but with random choices of predictors).”
That’s the head scratcher for me. Treemometers were originally assumed to comprise a random forest in the sense of being individual “models” of past temperature plus noise that could be averaged out to improve accuracy. But it didn’t work out that way. Instead, beyond a certain point, as new trees are added to the composite, the total variance decreases, and the output curve approaches a flat line. This phenomenon has been given the wonderfully literal name of “variance loss.”
I have no doubt that trees record genuine climate signal and some individual trees show excellent fidelity to regional temperature. But this variance loss has me questioning every other assumption about forming composites from noisy data.
> Instead, beyond a certain point, as new trees are added to the composite, the total variance decreases, and the output curve approaches a flat line.
If you’re just “averaging” then this is totally expected behavior right? Concentration of measure means it HAS to do that (a kind of generalization of the central limit theorem type idea).
If instead you’re averaging a small number of species but which species changes under different circumstances then you can get more variation.
“If you’re just “averaging” then this is totally expected behavior right? Concentration of measure means it HAS to do that (a kind of generalization of the central limit theorem type idea).”
If you were just trying to find the average tree ring width in the forest, then the trees would constitute a “random forest” and the more trees you measure, the better your estimate of average width. The measure converges on the true value as n increases.
But when we try to model a secondary variable such as climate from ring width, if we start with the best “model” tree and work down the list, we plateau at a certain point – maybe even after the first tree – and then our fidelity to reality declines as n increases because we are adding crappy treemometers that only serve to crush variation. The measure does not converge on the true value as n increases.
So with polling, does the composite average improve like our average ring width measurement when n increases, or does it plateau at one or a few polls and then just converge towards 50-50 when you add more polls, making it less accurate?
In the previous thread on polling, commenter “Paul” asked whether we might be better off using a single gold standard poll rather than a composite, I think he was getting at the same question.
No doubt others have thought much more than I have about this, so I’ll be reading the papers Andrew linked so I can learn.
Yes, the problem comes when the “bad” kind of trees are not just noisy but biased. If you continue to add tree species assuming they are unbiased, averaging will just converge with small standard deviation on the average bias across the species. You’ll get a constant that’s wrong.
If you assume a bias, and have some way to estimate it, you should be able to do better by including that in the model.
y = (a(x)-ba) + (b(x) – bb) + (c(x) – cc)
In the polling problem the issue appears to be that people think “averaging polls will get us the right answer” rather than “averaging corrected polls will get us the right answer”.
They’re implicitly assuming bias is zero rather than trying to subtract a bias and then use the fluctuations around that bias to learn something about the underlying construct.
For Rasmussen polls, the first thing I’d probably try is to incorporate a bias in your model: (R-rbias) would be zero centered if rbias was well estimated, and could be informative.
“For Rasmussen polls, the first thing I’d probably try is to incorporate a bias in your model: (R-rbias) would be zero centered if rbias was well estimated, and could be informative.
You know there is a thumb on the scale, so you push down on the other side…a bit. Not sure that qualifies as informative.
Nate Silver wrote:
“Rasmussen has had a Republican bias, but other polls have had a Democratic bias, so they come out of the wash looking OK? Yeah, probably.”
I am arguing that while the biases may approximately even out, the price for these kinds of shenanigans – whether we are adding bias adjustments or not – is a blurring of the signal in the direction of variance loss.
Matt. The idea is that if many polls shift a certain direction, then this is probably information that is valuable regardless of what the overall level is for any individual poll.
Perhaps Rasmussen is biased, but maybe if you have 4 polls and they all gain around a point for Dems, then the real opinion did shift a point-ish towards Dems, even if Rasmussen shows Repub leading by 5 points while everyone else shows neck and neck.
Similarly, if a certain tree tends to grow faster when its warmer, then when a certain year has a bigger ring than last year, probably the overall average temp was higher… Even if overall during a certain era this type of tree was always growing faster than other types of trees. Alternatively you might smooth through a few decades and at least say that the derivative of this smoothed function is perhaps indicative.
If you have a time-series and you ignore the time-series structure and look at a given point in time, and try to estimate from the polls with some biases, then you’re screwed. If you do use the time-series structure, you are much more likely to get somewhere.
@Matt Skaggs
On this particular topic, Rasmussen has been polling for years, and we have the final outcome of previous elections which they polled for. Assuming the bias in their methodology is consistent across elections, you can apply a bias correction to increase your effective sample and reduce the variance of your forecast. In your analogy, it’s like you have a limited number of noisy scales to use, and you know pretty much how much their thumb weighs, so it’d be costly to throw their scale out.
Now, as you point out, including every poll you can find anywhere on the internet would probably make your forecast worse rather than better; you have to be able to make reasonable assumptions about their sampling methodology, like that the bias in their methodology is consistent across elections. The point Morris is getting at, perhaps a bit tersely, in his letter, is that there has been a large scale disinformation campaign to make it seem like Democratic victories are illegitimate and inflate the perceived popularity of Republican candidates. In addition, Rasmussen is closely affiliated with important figures in this campaign. If Rasmussen has a conflicted interest, especially a new one, the assumption of consistency breaks down and their data go from a biased sample to effectively garbage.
And yeah, statements like this
are just nonsense. I’m honestly a little surprised — based on the methodological description of FiveThirtyEight’s forecasts under Nathaniel’s tenure, to the extent that they’re publicized, it should be obvious to Nathaniel that this reasoning is idiotic.
Thanks “somebody,” I 100% agree with your synopsis. Yes, if a pollster is consistently biased because they mostly sample college students or whatever, that is still useful data as long as the bias is adjusted for. But if instead a pollster decides to try differently worded questions until they get anti-abortion at 38% and pro-abortion at 37%, and then in an election they manipulate until they have Trump +5, there is no meaningful way to adjust for the house effect even though you can generate a direction and average magnitude for the bias just like you can for the college student problem. So I think what Elliot is doing is right as long as he looks for the same problems on the other side.
Dale:
I recommend these two articles:
[2018] Using stacking to average Bayesian predictive distributions (with discussion). {\em Bayesian Analysis} {\bf 13}, 917–1003. (Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman)
[2022] Bayesian hierarchical stacking: Some models are (somewhere) useful. {\em Bayesian Analysis} {\bf 17}, 1043–1071. (Yuling Yao, Gregor Pirš, Aki Vehtari, and Andrew Gelman)
Matt Skaggs wrote
“But if instead a pollster decides to try differently worded questions until they get anti-abortion at 38% and pro-abortion at 37%, and then…”
One of the triumphs of the extreme right is the wording of this issue. No one, and I repeat, no one, is pro-abortion. The proper wording, according to some, should be pro-abortion rights vs. anti-abortion servitude. Less clumsy, and perhaps more accurately, is female bodily autonomy vs. forced birthing. It should be noted that part of R.A. Fisher’s genius was his ability with words, such as “statistically significant,” “eugenics” and “(Fisher’s) exact test,” (which I believe Andrew holds in low esteem), so that choosing a definition can effectively tilt the table.
Is it just me or is there something unsettling about G-El being “brought in” to work on 538 forecasts? I like Eliot’s work but feels like the world is big enough for both of them without ABC letting Nate go and pulling his arch-rival in his place. It’s honestly a bit humiliating to the profession in general that ABC is trying to jump on the new hot talent and discarding the old. Feels like they’re athletes more than statisticians.