Being in a commerical space (rather than an academic space), he rather HAS to keep his models secret. If others can just run his models, he loses traffic to his site. His primary goals is NOT to advanced our knowledge of good political modelling, and so traffic to his site takes a much higher priority.

But that leaves us not really knowing what he is doing. To my knowledge, there is no context in which he opens up to the deatils of his models to criticism by outsiders, not even small groups under some sort of NDAs (formal or otherwise).

This makes it harder to learn from his work, harder for him to learrn from others, and leaves him more ignorant than he might be otherwise. That ignorance appears to be on display here, and it matters because of the prominence of his platform.

]]>Agree with these points, Andrew. Honestly this is all the post needs to be.. What is Nate talking about? No one even knows at this point.

]]>Thanks. Regarding the second point, I think my issue was with the formulation “if our stated probabilities do wind up diverging from the results” making a reference to “stated probabilities” rather than “predictions” or “forecasts”. It was not clear to me that “stated probabilities” meant “forecast of vote share” and I thought it was related to the probability of some outcome.

A few comments about the 95% confidence intervals:

The central forecast and the upper and lower bounds between today and election day wiggle up and down. I think it would be better to smooth those lines, otherwise you’re committing the graphical version of going against the “thou shalt not write numbers with non-sensical precision” commandment.

It seems that the width also varies, apparently due in part to the meaningless noise introduced by the simulation, but overall it is

surprisingly stable. The width seems to increase slightly first and decrease later, but it is not clear.

If the model “pulls” the forecast to some “fundamental” forecast one could expect that the probability mass above 57% for Biden would be pulled down while the probability mass below 53% would be pulled up (assuming the “target” is within that range, as it seems from the chart).

If paths are thus attracted to some “forecast based on non-polling data” the probability would concentrate. On the other hand, uncertainty should increase as we go further. Maybe both “narrowing” and “widening” are present in the model and they offset each other?

If, according to the model, say that

a) there is 95% probability that support for Biden tomorrow is in the 51.8%-57.8% range

b) there is 95% probability that support for Biden on October 18 is in the 51.4%-57.4% range

Do you believe those probabilities to the same extent? Do you find that the model is equally reasonable at the one-day and the three-months horizon? If you had to bet on one of those statements (assuming the true support at the time could be determined to settle the bet) would you be indifferent between them?

]]>Anon:

See my comment to Michael here. I have the impression that Nate tried some partial pooling himself and it didn’t work well, and he’s attributing the failure to the method rather than his implementation of it. I wrote this post because Nate has a lot of readers, and I’m concerned that they’d read his post and think there’s something wrong with MRP.

But, yeah, MRP, like any sophisticated method, can screw up, and more research is needed on understanding these sorts of problems.

]]>Michael:

I think what happened is that Nate has tried some partial pooling and it hasn’t worked so well for him. That’s why I invited him to share his code, as then maybe people could point out how he could do better.

Statistical methods can be tricky, and it’s easy to mess up if you try to implement them from scratch. Just for example, I’ve read about deep learning and how cool it is.

But if I tried to fit a deep learning model on my own, I’m sure there are all sorts of ways I could go wrong, all sorts of tacit knowledge that I’d not be including that would lead to me getting bad results. The problem wouldn’t be with deep learning, it would be with my implementation of deep learning.

Now, at this point, you could criticize MRP, or deep learning, or some other cutting-edge method, as not being fully mature, in the sense that you can’t just push a button, you have to be careful in setting up your problem or you can get bad results. And that would be a valid criticism. But that’s just the way it is with new methods. It takes awhile to set them up so that they are easy for outsiders to use.

To return to a point that has come up over and over again on this blog: none of us is an island when it comes to statistical methodology, and any of us can be made stronger by sharing our methods and code and opening ourselves up to criticism. By hiding behind his twitter wall, Nate’s not making use of that opportunity. If he were to share his code, he could do better. It’s too bad, really.

]]>Joshua:

Yes, there are many potential interactions; see this article. No model will capture everything, so in practice we think of these as average adjustments, for example adjusting for average differences between Hispanics and non-Hispanic whites of the same sex, age, and education categories. Nate’s comment was missing the point that this adjustment does *not* assume that the voting patterns of 46-year-old Hispanic men are the same in every state. Again, this is not to say that regression models are perfect. But, as we saw in 2016, raw data are not so perfect either.

I would guess that comparisons among Hispanic males with *different* demographic profiles in the same state will also vary state by state, also. IOW, maybe age differences are more explanatory in one state whereas education differences are in another. If so, how do you weight different parameters (e.g., ethnicity vs.age vs.income vs. education level) in a way that allows for an appropriate variance in line with one state vs. the next state? Or do you just assume it works OK to weight those variables relative to each other in the same way across different states?

]]>Carlos:

The uncertainty ranges are 95% posterior probabilities for support for each of the two candidates at each day. It’s all conditional on the model. Regarding, “what does it mean for the results and your stated probabilities to diverge,” see Figure 2 of this paper.

]]>Occasional:

This is the point of cooperation, not competition. No need to use state polls *or* national polls. We can use both!

The article ends with: “And if our stated probabilities do wind up diverging from the results, we will welcome the opportunity to learn from our mistakes and do better next time.” What does it mean for the results and your stated probabilities to diverge?

[The link to the forecast brought this question it to my mind again, but I understand it is not as relevant to this post as it was to the June 19 post.]

]]>