No, Bayes does not like Mayor Pete. (Pitfalls of using implied betting market odds to estimate electability.)

Asher Meir points to this amusing post from Greg Mankiw, who writes:

Who has the best chance of beating Donald Trump? A clue can be found using Bayes Theorem.

Here is the logic. Let A be the event that a candidate wins the general election, and B be the event that a candidate wins his or her party’s nomination. Predictit gives us the betting market’s view of P(A) and P(B). It is a safe assumption that P(B|A) = 1, that is, a candidate can win only if nominated. We can then use Bayes theorem to compute P(A|B), the probability that the candidate will win the general election conditional on being nominated.

So here are the results for P(A|B) as of now:

Buttigieg 0.80
Biden 0.77
O’Rourke 0.67
Sanders 0.65
Booker 0.60
Yang 0.60
Harris 0.57
Warren 0.44

That is, the betting markets suggest that Mayor Pete would be the strongest candidate if nominated, with Joe Biden close behind. (Of course, these numbers will bounce around as the prices in betting markets change.)

By the way, when I [Mankiw] did a similar calculation in 2006, Bayes liked Barack Obama.

I copied Mankiw’s post in its entirety, with the only change being that he wrote P(A / B) etc., and I changed the slash to the vertical bar, P(A|B). (Are there people who write conditioning using a slash rather than a vertical bar? I had no idea. P(A|B) is more standard, I believe. In the above post, Mankiw links to the wikipedia page which uses the P(A|B) notation. No big deal, it just seemed odd to me.)

Anyway, I think the above set of calculations is a great example for teaching conditional probability.

The next step is to push a bit: Do we really believe these numbers? There’s nothing wrong with the probability calculations, but I’m not sure we should be taking Predictit’s betting odds as actual win probabilities.

To start with, I looked at Mankiw’s list and wondered what Yang was doing on it. Yang’s a fringe candidate, right? He’s polling at 0.8% on Real Clear Politics right now. I went over to Predictit and it says you can buy a Yang contract for the Democratic nomination for the price of 9%. Now, OK, sure, at 0.8% in the polls there’s room for improvement. But 9%??? Seems like a lot.

I know I’m leaving a hostage to fortune here, writing this now in June and scheduling the post for the end of November. If by that time Yang is a serious candidate I’ll add a P.S. to this post.

The next think I’m worried about, beyond bias in the online markets, is volatility.

Sure, Mankiw writes, “these numbers will bounce around as the prices in betting markets change,” but I think he’s not fully appreciating how noisy these numbers are!

Mankiw’s post is dated 27 Apr 2019. Predictit conveniently gives prices going back a few months, so I can do some Biden-Warren price comparisons of then to now:

27 Apr 12 Jun
Biden primary election 22 28
Biden general election 17 19
Warren primary election 9 19
Warren general election 4 13

Something weird was going on in April, when Biden’s price was 22 for the primary and 17 for the general election. This just can’t be right, and all I can conclude is that the betting markets here are thin enough that nobody was taking these numbers very seriously.

If you want to take the numbers as is, you’ll get the following:

27 Apr: Biden 17/22 = 0.77, Warren 4/9 = 0.44
12 Jun: Biden 19/28 = 0.68, Warren 13/19 = 0.68.

These numbers aren’t quite right, even if you take these betting markets seriously, because of rounding and the vig. If you add up all the prices on the “Who will win the 2020 Democratic presidential nomination?” page, you get something well over 100%. So you can’t directly interpret these prices as probabilities, even beyond the issues of bias and noise.

I discussed this with David Rothschild, who thinks a lot about elections and prediction markets (for example, here), and David responded as follows:

People ask me to compute this automatically on my blog, but I refrain, because it is so noisy this early. Here I compute the conditional probability range separately for Betfair and PredictIt, by diving the seller’s price of win / buyer’s price of nom & buyer’s price of win / seller’s price of nom. Betfair has advantage of being tighter by definition (PredictIt trades on the penny, but Betfair on the odds, which have more depth).

Here is a figure from an old paper I wrote with David Pennock about the 2012 election. As you can see, while informative, it can get quite noisy!

Anyway, my point here is not to criticize Mankiw but rather to thank him for putting out this fun example, and then to demonstrate how we can take it further by interrogating each step in the analysis. Which is how we do applied statistics in general.

P.S. In case you’re curious, based on the numbers today, where Biden’s implied electability is 19/28 = 0.68 and Warren’s is 13/19 = 0.68, we can look up Buttigieg. He’s at 9/16 = 0.56, the least electable of the three. So, no, Bayes does not like Mayor Pete.

It’s a fun example, but when we look at the data more carefully, the original conclusion goes away.

122 thoughts on “No, Bayes does not like Mayor Pete. (Pitfalls of using implied betting market odds to estimate electability.)

  1. I trade these markets fairly often and usually most inconsistencies you see in prices across markets are more a reflection of the market being inefficient than it having any particular insight. For example, the prices of the democratic nominees consistently adding up to 1.15+ (that even increases any time a new candidate gets added to the market!). So even the implied probabilities from a single market are not reliable, let alone from the combination of multiple markets.

  2. Quoted from a comment I previously posted on another blog:

    “ Important to understand that P(Candidate X Wins | Candidate X Nominated) is not the same thing as P(Candidate X Wins | do(Candidate X Nominated)), to use Pearl’s notation. In English, the probability that a candidate wins given that we *observe* them winning the nomination is different from the probability that a candidate wins given that we *intervene* so that they win the nomination.

    Take Yang for example. The first of these probabilities is 0.034 / 0.049 = 0.694. One might conclude that he is therefore very electable, and that we should all coalesce around him so that the Dems win back the White House. In fact, what this probability really means is that *in the type of world where Yang becomes the nominee, he is also likely to win the election*. In other words, if something crazy happens, causing Yang to win the nomination, it is also likely to cause him to win the general election. On the other hand, if we infer electability from this probability, and thereby make him the nominee, this crazy thing wouldn’t have happened, so there’s no reason to think he’d be in a strong position to win the general election.

    Macroeconomists should recognize this as a generalization of the Lucas Critique. The association between winning a nomination and winning the general *under observation* is not the same as the association between winning a nomination and winning the general *under intervention*. For the electability question, what we really want is the second probability, but this cannot directly be computed from the listed prices. We would need to observe plausibly exogenous variation in a candidate’s chances of winning the nomination, and see how their probability of winning the general responds, and daily fluctuations in these things are not exogenous.

    I realize this is not a point being made in the post, but I see this confusion a lot. While it’s true that these markets are thin, particularly for the long-shot candidates, the other reason their conditional probabilities may seem wacky is that it’s only in wacky scenarios that they become the nominee in the first place. In such a wacky scenario, it wouldn’t be all that much more wacky for them to go on to win the White House.”

    Source: https://www.themoneyillusion.com/conditional-probability-and-election-odds/#comment-5194735

    • If a candidate successfully convinced votes that he was the only most likely candidate to win the general election by manifesting P(Candidate X Wins | Candidate X Nominated), would such campaign strategy belong to “something crazy” that causes him to win the nomination and therefore also more likely in the general election? I cannot image any situation in which we can identify the treatment effect of X’s nomination (conditioning on the “do” operator).

    • Ram:

      You’re making a mistake here. No causal inference is required here. It’s just simple conditional probability. Yes, conditional on Yang being nominated, an unlikely event has happened. That unlikeliness should be reflected in the probability he wins the general election. Intervention or observation has nothing to do with it. To the extent that the probabilities reflect actual betting beliefs, they should correspond to some joint distribution that accounts for all the probabilities.

      • P(Candidate X Wins | Candidate X Nominated) is conditional on “Candidate X Nominated” only. The probability conditional on “Candidate X Nominated Under Circumstances Y” is a not-so-simple conditional probability.

        P(X is rich | X drives a Lexus) is high. P(X is rich | X drives a Lexus he bought to become rich because he noticed the correlation between dough and ride) not so much.

        P(Candidate X Wins | Candidate X Nominated) may well be higher than P(Candidate X Wins | Candidate X Nominated Because Of The Wrong Reasons).

        • Another way to say this is when the nomination comes along, there is a lot more information available than that he just was nominated. There is no one thing called “nominated” or even “do(nominated)”… these are composite things

        • Carlos:

          Sure, but you’re complicating things. Again, if the odds could really be taken as coherent betting probabilities (which they can’t, for reasons including the vig and the thinness of the market), then it’s just Pr(A&B) = Pr(A|B)*Pr(B). The fact that you can break these down into C, D, E, etc., is substantively interesting and potentially relevant when deciding how to set the odds. But once the odds are there, they directly apply to A and B, and they are implicitly integrating out C, D, E, etc.

        • I think that the point that Ram is making is that even though P(Candidate X Wins | Candidate X Nominated)=0.7 this doesn’t necessarily mean that if all the other candidates withdraw from the race P(Candidate X Wins)=0.7 and I fully agree.

          Another betting example (I don’t really know how the NBA works, I hope it makes sense): You may have a model that gives you a completely coherent set of predictions.

          Say the Sacramento Kings have according to your model some probability of playing the finals and some probability of winning. We agree that you can calculate, for example, P(Kings win the championship | Kings play the finals)=55%.

          If you now bribe all their rivals in the Western Conference to let the Sacramento Kings proceed to the finals regardless of their performance, would you say that the original calculation remains valid and P(Kings win the championship)=55% ?

        • “If you now bribe all their rivals in the Western Conference [is] P…=55%” still valid?

          This really clarifies the distinction between “under observation” and “under interference”. thanks!

          I find the term “under observation” somewhat odd and ill fitting. What about “unaffected” or “natural” or “with / without intervention” or something like that?

      • Andrew,

        I’m not sure what mistake you think I’m making. I understand that you can calculate P(Win | Nominated) from P(Win) and P(Nominated), if you assume P(Nominated | Win) = 1. The point I’m making is that the concept of electability is not captured by P(Win | Nominated).

        If my objective is *predicting* whether a candidate will win given they secure the nomination, P(Win | Nominated) is precisely what I want to know. If my objective is instead *intervening* to hand the nomination to the most electable candidate, P(Win | Nominated) is not what I’m looking for.

        E.g., suppose the only scenario with positive probability *under observation* in which Yang wins the nomination involves Trump signing a contract saying he will run against Yang but withdraw at the last minute to hand him the general election. Then if Yang in fact is nominated, that means this contract was signed, and it is therefore very likely that Yang will win the general election. Bettors would be right to price P(Win) and P(Nominated) in such a way that their ratio is close to 1. And yet it would be preposterous to say this means Yang is electable.

        If we did conclude this, we might have a bunch of folks work to nominate Yang, which *changes the underlying distribution*. After all, now there is another positive probability scenario in which he gets the nomination, and so P(Win | Nominated) could well drop to 0! Did the realization that he’s the most electable candidate immediately cause him to become completely unelectable? Or was this conditional probability never measuring electability to begin with?

        • Ram:

          You are wrong. Th econcept of electability is exactly captured by P(win|nominated). Nobody is intervening here; we’re talking about bets on the outcome. In your Yang example, this scenario would already be included as one of the many many different possible ways of him winning. Yes, we have a bunch of folks working to nominate Yang, as we do all the other candidates. The betting odds are supposed to represent the marginal probabilities, integrating out all the many different things that people will do in the future.

          I’m assuming that betting a few bucks in the prediction market is not itself an intervention that can affect a candidate’s probability of winning. I agree that if it’s possible to lay down such a big bet that it actually changes the odds, that’s another story entirely. (Supposedly there was an ineffectual attempt to do this in 2012.)

        • Andrew,

          Last try, then I’ll agree to disagree. Suppose in my Yang example that, even though a large proportion of Democratic primary voters would like to vote for the most electable candidate, few or none understand your view that electability = P(Win | Nominated). As a result, no one acts on the fact that, in my example, P(Win | Nominated) ~= 1 for Yang. Would you then say that in fact Yang is the most electable candidate, but that if voters were to be appropriately educated about the correct definition of electability, he would immediately become completely unelectable?

        • Ram:

          There are two things going on here.

          1. If the implied probabilities from the betting markets are subjective betting probabilities, then they should obey the laws of probability, so that we can calculate conditional probabilities from appropriate marginal and joint probabilities. I agree with Mankiw there. Where I disagree is with Mankiw’s assumption that the betting market odds can be taken to imply subjective betting probabilities. I think the market is too thin, too noisy, and has too many biases for this to work.

          2. “Electability” is just a word. It means different things to different people. As a statistician, I think “electability” is P(win|nominated), nothing more, nothing less. But I recognize that people use the word in other ways. I just googled it and came up with this article and this article.

        • Andrew,

          1. Agree completely, I don’t think I said anything differently.
          2. Sure, we can define words however we like. What I’m wondering is why anyone should care about P(Win | Nominated). My claim is that some people (e.g., Mankiw) care about this quantity because they think it is useful for identifying candidates who will fare better in the general election. My point is that this is fallacious. Just because a candidate is likely to win the general in the same possible worlds where they win the nomination, does not mean that the party would be well-served by nominating this candidate.

        • Ram:

          Your claim #2 is incorrect. One way to see this is to use the trick of converting probabilities to frequencies. Suppose a candidate has a 10% chance of winning the nomination and a 6% chance of winning the general election. Then imagine the entire process being run forward in 1000 different worlds. In 100 of these worlds, the candidate would be the nominee. In 60 of these worlds, he’d win the election. It is what it is. These numbers automatically account for whatever it would take for the nomination to happen. Again, if the bet itself can have an effect on the election, that’s another story.

        • I think we can infer the state of the world is very different in different scenarios where different people are nominated.

          For example we might say “Yang can only be nominated if everyone else has been assassinated in a crazy plot”… So p(everyone else dead | Yang nominated) = 1 and p(Yang wins | yang nominated) is equal to p(Yang wins | everyone else is dead).

          On the other hand, if say Biden is nominated we couldn’t infer that the world is a lot different than it is today… since Biden nominated is not a particularly outlandish scenario for worlds similar to today’s world.

          On the other hand, if the convention looks at the numbers on the betting things, and says to themselves “hey let’s nominate Yang, he’s got about a 100% chance to win if he’s nominated”… the state of the world in which he’s nominated is not the one that the model assumed was necessary… so the day after he’s nominated, we have new information, and p(Yang wins|nominated) drops to near 0.

          This is yet again an issue with mind projection fallacy… people treat p(Wins | nominated) as an objective property of the universe, like the shape and balance of a particular die, rather than a property of the information people have about the world.

        • Seems like everyone is basically understanding what I’m saying besides Andrew, so I think I’ll call it a day. I agree with the various other comments:

          Yuling is concerned about the ambiguity of P(Win | do(Nominated)). I agree this is ambiguous as a general concept, but I think when people are concerned about electability, they’re concerned about some particular way of disambiguating this quantity. What they’re not concerned about, I’m arguing, is the observational conditional P(Win | Nominated). That this second quantity is unambiguous doesn’t mean it’s the thing people care about when they talk about electability.

          Daniel is right that the issue here is that the joint distribution that Andrew keeps referring to changes when we intervene upon the system it is describing. This is the point of distinguishing between P(Y | X) and P(Y | do(X)). The former is a quantity derived from the joint distribution of X and Y that characterized the system under the observation. When we intervene on X, we change this system, and so change the joint distribution of X and Y that characterizes it. Hence any conditional probabilities derived from this joint distribution change as well. My point is that when people talk electability, they’re talking about what this conditional looks like AFTER intervention on the joint distribution, not before. The quantity Andrew is describing as electability is the BEFORE intervention conditional.

          Carlos seems to know exactly what I’m talking about: even if the conditional probability of the Kings winning the championship given they make the playoffs is 55%, this tells us nothing about how they would do if the league rigged all the regular season games to ensure they make the playoffs. Why? Because this intervention would change the relative frequencies of different playoff events, and so the 55% is a before-intervention quantity, when what we want to here is the after-intervention quantity.

        • Ram:

          Nope, you’re wrong. I don’t have the energy to try to explain further here, but I guess it’s good to have uncovered this particular point of confusion. Recall that it took years before people could come up with a clear explanation of the Monty Hall problem. Probability is tricky, and people can easily get confused! I tried my best with the “run 1000 elections” scenario, but I guess that wasn’t enough….

        • Andrew,

          Thanks for responding to each post. Happy to stop here. Since you keep saying I’m wrong/mistaken/incorrect, I’ll just note the structure of my argument is this:

          (1) In general, P(Win | Nominated) != P(Win | do(Nominated))
          (2) When people are trying to figure out which candidates are more or less electable, what they’re usually after is some particular disambiguation of P(Win | do(Nominated))
          (3) Efficient prediction markets for who wins the election, and who wins the nomination, can be used to estimate P(Win) and P(Nominated)
          (4) P(Nominated | Win) = 1

          Ergo

          (5) Even ignoring issues of market inefficiency, we cannot in general estimate electability using ratios of these prediction market prices.

          I’m curious which of (1)-(4) you’re objecting to, or whether you’re objecting to the inference leading to (5). I’m also curious why you have not said Carlos is wrong, but merely complicating things, when his examples are just particular illustrations of this argument.

          In any event, thanks again for your responses. Over and out.

        • Ram:

          In response:

          (1) is irrelevant. There’s no such thing as “do(Nominated)” anyway, but that doesn’t even matter because all that matters here are the probabilities.

          (2) I have no idea what people are trying to figure out. You might be right about the psychology, hence you might be on to some sort of explanation as to why the implied betting odds don’t make sense.

          (3, 4) Yes.

          If the betting markets are efficient then we should be able to extract all the joint and conditional probabilities of interest. The betting markets clearly are not efficient, given that the probabilities don’t make sense and are not coherent over time (as discussed in my above post). It might be that bettors’ misunderstanding of conditional probability and a confusion with causation could be contributing to the inefficiency of the betting markets.

        • It does seem like everyone but Andrew is understanding this (and, perhaps ironically, he has the strongest conviction that he is *correct* here)

          To continue with the earlier basketball example, consider a team that is very unlikely to win the NBA title at the start of the season. Let’s say their *true* skill level is a 2.5 on some scale from 1-10. Further, let’s suppose that *true* skill varies over the course of a season (based on wins and losses). Define true however you like; please don’t hijack this comment with a debate about the meaning of *true* probabilities.

          Now, if this team did happen to make it to the NBA final, it is very likely that their skill level will be much higher than 2.5 at the time of the Finals (because to get to the NBA final they would have had to be higher skill than their start-of-season skill, or simply been very lucky). Therefore, their probability of winning conditional on getting to the NBA final will be much higher than the win probability of a 2.5 skill team against the average opponent one can be expected to face in the NBA final.

          This seems directly analogous to the Yang electability question. Currently Yang is a “low-skill” candidate, and if he were to face Trump in an election tomorrow would (perhaps) lose by a landslide. But, in the worlds where Yang goes on to win the primary, he will likely be a ‘high-skill’ candidate in those worlds (for any number of reasons). Therefore the conditional probability will be much greater than the probability of Yang winning an election with Trump tomorrow (at his current, low skill).

        • Matt:

          It’s amusing, all right, reminiscent of the Monty Hall thing from many years ago. I expect it’s mostly a communication problem. Maybe I can enlist Josh Miller to explain it to all of you. The relevant econ buzzword is “reduced form”: we just care about Pr(X,Y), where these probabilities average over all uncertainties. Given that this is not yet clear to people, I’ll have to let someone other than me explain it.

        • Andrew,

          Yes I think it is just communication, and reduced-form does seem like a relevant buzzword.

          However… what if P(Yang win general | Yang win candidacy) already accounts for the fact that tweets like Mankiw’s would occur, and then people would start voting in Yang on the basis of his perceived *electability* (as inferred from Mankiw’s conditional probability calculation). Therefore this would mean that the conditional probability already accounts for the fact that some of the time Yang gets elected it is exactly because people _think_ he has high electability (which they inferred from the current betting odds as Mankiw did). But then this should put downward pressure on the conditional probability (i.e. the denominator will get bigger).. which will then cause people to stop voting Yang in on the basis of inferred electability from current market odds! And around and around we go… until we stabilize at an equilibrium!

          Wow, this is absolute nonsense. Best to post it anyways; I assume someone will get a laugh.

        • I’m with Andrew on this one.

          The anti-Andrew crowd is assuming some sort of meta-game where actors outside the electoral process can intervene to affect the nomination (a big donor, massive assassination plot, or whatever). If these meta-actors use probabilities derived from the non-meta world, then they will be in error because they are ignoring the meta-world … electability then depends not only on real-world information, but meta-world information as well.

          Andrew only wants to look at the real world. In the real world, people are thinking about electability and will vote based on their thoughts about electability, but that is all in the betting odds … *everything* in the real world is in the betting odds. Andrew doesn’t care about the meta-game because the meta-game is not real. While it might be fun to think about a meta-game, that is not the real world.

        • Terry,

          What do you make of Matt’s basketball example? Translated to politics, it would go something like this:

          Each candidate has a favorability rating with the voting public as a whole, which varies over the course of election season. (For simplicity, assume that the best estimate of future favorability is present favorability.) Most voters, upon learning a candidate has very low favorability, would conclude that the candidate is “not electable”, meaning that a candidate with such low favorability is unlikely to perform well in the general election.

          Nevertheless, in virtually any scenario in which the candidate *does* secure the nomination, this would have been preceded by a sharp rise in their favorability (since candidates with too-low favorability rarely get nominated). As such, conditional on being nominated, the candidate is reasonably likely to win the election.

          If we take P(Win | Nominated) to be a measure of electability, this candidate would be found to be reasonably electable, despite voters agreeing that the candidate is not electable. Does this not suggest something is amiss with this measure of electability?

        • Ram:

          I have no problem with the basketball example. Seems right to me.

          “Electability” is a time varying state variable. No reason Yang’s can’t go way up to the point where he gets nominated, and Yang will only be nominated if he becomes more popular/electable. The calculated probabilities of winning given nominated are all pretty high which makes sense to me.

        • I think the problem is that “electability” is seen as an important consideration in whether someone is nominated… Nominate the “most electable”… if p(elected | nomination) *is* electability then one should nominate the person with the highest value for that… of course, if we all realize that what’s happening is that the person with the highest p(elected | nomination) is going to be nominated, we will use that information to change the estimate of p(elected | nomination), and we should expect it to crash for less supported candidates…

        • It helps a lot to use better notation:

          P(elected | nominated)

          should be written

          P(elected | nominated, BackgroundKnowledge)

          Now, obviously the quantity changes when the BackgroundKnowledge changes. One way this background knowledge can change is that we can know that people are now putting more support on nomination of this candidate for any reason at all, including that the price of certain contracts is high or that fancy financier is throwing money into the person’s campaign, or any number of other things.

          As of today, the market prices may reflect what people think is likely, but if people use the contract prices to alter the nomination plans, then also people will use the new nomination plans to alter the contract prices as well… leading to a dynamic process where the prices of the contracts could change a lot, including oscillate.

          I think Andrew’s take on it is: all of this is supposed to be priced in, and the price is already in equilibrium… except he doesn’t believe that of course, but if it were true… it would be true, and p(elected | nominated) would be the relevant probability.

          I think the rest of us see that obviously the prices *aren’t* all priced out in 4D chess 1000 moves ahead… and so to the extent that we start publicizing the idea of p(elected | nominated) as “electability” we should expect the contract prices to change a bunch leading to p(elected | nominated) collapsing for more fringe candidates, at which point… they would be closer to the equilibrium Andrew was discussing and yes, at that later equilibriated time p(elected | nominated) would be the relevant probability.

          However, to the extent that the p(elected|nominated) is not being used to decide on the nomination… todays prices may well be moderately efficient prices giving reasonable estimates, because the feedback mechanism which requires much longer to equilibriate, is not in play.

        • I think Andrew is even more right than he realizes.

          Pr(Win|Nominated) is not just an abstract calculation. It is a contract that you can construct right now from the existing PredictIt contracts for the price Andrew calculates (ignoring market inefficiencies.)

          Pr(Win|Nominated) is a forward contract. In a forward contract, you agree to the terms of the transaction today, and you agree to the price today, but you execute the transaction sometime in the future. The important point is that everything (price, terms, and transaction date) are locked in *today* so you know everything today and there is no uncertainty. The price of the forward contract is calculated with mathematical certainty because it can be constructed from the contracts available today, which means the price of the forward contract is the sum of the prices of the contracts you have to buy to construct it. In the forward contract, you would agree that, if Buttigieg is nominated, you will buy a contract that pays $1 if Buttigieg wins the election and you will pay the seller of the contract $0.80. (I’m a little fuzzy on exactly how to construct the forward because it is conditional on winning the nomination.)

        • Daniel said:

          I think the problem is that “electability” is seen as an important consideration in whether someone is nominated… Nominate the “most electable”… if p(elected | nomination) *is* electability then one should nominate the person with the highest value for that… of course, if we all realize that what’s happening is that the person with the highest p(elected | nomination) is going to be nominated, we will use that information to change the estimate of p(elected | nomination), and we should expect it to crash for less supported candidates…

          That’s why I put quotes around “electability” and later referred to “popular/electible”. It isn’t purely considerations of electability. It is a mish mosh of many thing of which electability is probably not even the most important. These factors interact in many unknowable ways, one of which might be a feedback loop about electability. I don’t care. That is just another way of saying it is complicated. All I know is the market chewed on a lot of things and arrived at these prices which we can interpret as probabilities. Given all that, Andrew is correct in how to calculate Pr(Win|Nominated).

          You seem to kind of agree with this line of reasoning in your last paragraph, so we may not even be disagreeing.

        • The price of the contract is whatever it is… whether those prices are accurate assessments of probability of things occurring based on everything we know today is another story.

          Suppose at the moment the price of a contract which pays $1 at the end of the day tomorrow is $0.1, you can either conclude that there’s a 90% chance that the world will end before tomorrow and people are just trying to gather as much money as possible today so they can have a blowout end of the world bash…

          or that someone hasn’t really thought things through and is willing to give you about a 1000% / day return on investment…

          When you go to purchase 1M of these contracts you will find that after buying about 4 of these contracts at $0.10 from Mr Som Dam Foo, the asking price for the remainder of the contracts has moved to $0.99997

          Andrew’s point could well be that *if* everything is priced in correctly, then p(elected | nominated) is all you need to figure out which person you should nominate, just pick the one with the highest p(elected | nominated)

          on the other hand, the rest of us think if someone starts taking this numerical concept seriously, then *everything isn’t priced in correctly* and you will rapidly see a dynamic change in the market prices once the fact that the nomination will be given to the person with the highest p(elected | nominated) becomes widely known.

        • Terry, I’m not sure whether we disagree either… but here’s a question. Do you think that if the Democratic party announced it would simply hand the nomination to whichever candidate had the highest contract price for p(elected|nominated) at the end of trading on Friday, that the price would

          a) move a lot by end of day
          b) stay basically the same throughout the day

          I vote a

        • Nice discussion.

          One thing I’d add is quibble with a point made by both Ram and Matt, which may address the communication problem Andrew is talking about.

          Ram wrote:

          Nevertheless, in virtually any scenario in which the candidate *does* secure the nomination, this would have been preceded by a sharp rise in their favorability (since candidates with too-low favorability rarely get nominated). As such, conditional on being nominated, the candidate is reasonably likely to win the election.

          If we take P(Win | Nominated) to be a measure of electability, this candidate would be found to be reasonably electable, despite voters agreeing that the candidate is not electable. Does this not suggest something is amiss with this measure of electability?

          I don’t see anything amiss. Of course, we all agree—from the beginning, I presume—that we can construct simple models in which P(Win | Nominated) is a terrible measure of electability. Carlos has a clean example that illustrates this point in which there is a single contingency Z that makes P(Win|Nominated, Z)=P(Nominated|Z)=1 (Z =war with Canada). I examples like this to be useful for our discussion because in practice there are many contingencies Z in which P(Win|Nominated,Z)≈P(Win|Nominated) but P(Nominated|Z) > > P(Nominated). The market probabilities average across these contingencies as well.

          Now, returning to your point:

          Let’s first assume that betting markets are operating as intended. Let’s assume that either (i) there is no feedback from market prices into fundamentals (e.g. the subjective beliefs of agents within the electoral process do not depend on market prices), or (ii) whatever feedback there is has stabilized market trading, and the beliefs and actions of agents within the electoral process (i.e. we are in a rational expectations equilibrium given the history up to this point in time).

          To keep it simple, let’s stick with (i) and assume that we are talking about the beliefs of agents from the outside looking in, what Terry refers to as “meta-actors,” but let’s keep them passive. Talking heads if you will. The claim is that P( i elected | i nominated) meaningfully represents electability.

          Because it is easy to get distracted by what we know about the USA, so let’s consider country A, with a similar electoral process and an incumbent waiting to face the nominee of the opposition party. Take two candidates i,j from the opposition party. Let’s assume that at time t betting market prices show that P_t( i elected | i nominated)=.90 and P_t(j elected | j nominated)=.10, yet P_t(j nominated)=.90 and P_t(i nominated) =.10. Can an outsider looking at these market probabilities learn anything about electability? I think so.

          The outsider has learned: (i) j is not electable in a general election, (ii) i would be electable in a general election if i could figure out how to become nominate-able. Obviously becoming more nominate-able at t+1, increases overall electability P_{t+1}(i elected), but that isn’t the object we are interested right now at time t. As Ram mentions, the conditional probability P_t( i elected | i nominated) includes the bump to electability that will arise after winning the nomination, which necessarily includes the automatic support of the opposition party voters. But the betting market accounts for more than this. Speculators are considering information about candidate attributes that play well in the general, but don’t play well in the primary and have to be overcome. They are betting on this information and this is reflected in the prices. That is why P_t( i elected | i nominated) = .90 > P_t(j elected | j nominated)=.10.

          I believe this is what Andrew was talking about when he called this electability.

          Perhaps some of the apparent disagreement we have had has to do with none of us stating our assumptions, which can generate miscommunication.

        • Daniel said:

          Terry, I’m not sure whether we disagree either… but here’s a question. Do you think that if the Democratic party announced it would simply hand the nomination to whichever candidate had the highest contract price for p(elected|nominated) at the end of trading on Friday, that the price would

          a) move a lot by end of day
          b) stay basically the same throughout the day

          I vote a

          I vote a) too for two reasons.

          One, since period T (the final period) has been moved up to Friday, one price should move to $1.00 while the others move to 0. There is no more time for new information, so future volatility goes to zero. Actual PredictIt prices go to 1 or 0 as election returns come in.

          Two, all the money usually spent by donors on campaigns would be spent in the PredictIt market so volatility would be enormous. As each new batch of money (billions perhaps) hit the market, a huge new shock would hit the market. I’m assuming PredictIt would continue to function. There could also be billionaires trying to nominate a bad candidate that would go down to defeat. Interesting gaming about whether to throw your money in early (first mover advantage) or late (to cause a stampede at the last minute).

          No idea how quickly consensus would develop around one candidate.

        • Joshua,

          > Obviously becoming more nominate-able at t+1, increases overall electability

          Not necessarily. One can easily imagine a scandal affecting the party that changes the nomination probabilities and at the same time reduces the probability of winning the election for all of them. (By the way, I was joking about the “contrivedness” of the examples.)

        • Terry,

          I think that everyone in the anti-Andrew crew agrees with him on how to calculate Pr(Win|Nominated).

          The disagreement, if any, was about the following claim:

          « the probability that a candidate wins given that we *observe* them winning the nomination is different from the probability that a candidate wins given that we *intervene* so that they win the nomination. »

        • Terry, I agree with the huge volatility. I also agree with the price of nominated going to 1 though I’m not sure it would get there by end of day, their might be a lot of oscillation at the last minute, it would open the next day at 1 for sure.

          Interestingly though we still don’t know about winning the election, so while the price of win would drop to 0 for all the non nominated candidates on open the next day, the price of win for the nominated candidate would not go to 1 or 0.

          How about this scenario, which involves less gaming… The Democratic party announces on Monday that it used Friday’s closing prices to decide who the nominee was… And chose whoever had highest p(win)/p(nominated).

          Obviously nominated contracts for the nominee go to 1 but do we expect p(win|nominated) to be unchanged? I still say no, it would change, and specifically if the candidate was an outsider it might change a lot, since p(win|nominated) for those candidates was based on scenarios other than the one where the Democratic party didn’t bother with a primary.

        • All,

          Forget about correlation v. causation, and observing v. intervening. While I find that connection illuminating, it seems to be confusing to others.

          What we mean by “electability” is how a candidate who has the characteristics she *actually has* would perform in the general election. What we do NOT mean by “electability” is how a candidate who has characteristics she *is predicted to have given that she was nominated* would perform in the general election.

          P(Win | Nominated) measures the latter, because in predicting the election outcome on the basis of nomination, it infers that certain characteristics of the candidate evolved in previously unanticipated ways, since nomination itself was not previously given. What we would really like to know is P(Win | Nominated, Actual Characteristics).

          An efficient betting market fully conditions on all publicly available information, so one might think that these actual characteristics have already been factored in as desired (if we pretend the market is efficient). However, many characteristics vary over the course of the primaries, or else only become public knowledge at some later point in primary season. Conditioning on nomination *changes* the best estimate of these as yet unknown characteristics, and thereby changes the probability of winning. What we really want is to condition on nomination as well as the *current* best estimate of these characteristics (this estimate itself not being influenced by nomination status). Forget about betting markets—even knowing the true P(Win | Nominated, All Public Information) would not give us what we want.

          As an example, suppose that the more left wing you are, ceteris paribus, the more likely you are to be nominated. However, the more moderate you are, the better your prospects in the general election. A relatively moderate candidate at time T is going to have a tough time getting nominated, but if we learn that they somehow *did* get nominated, we would infer that they must have moved to the left after T. On the basis of that inference, we would further infer that they’re gong to have a tough time winning, because they’re now on the far left. Thus, P(Win | Nominated, All Public Information at T) would be low for this candidate. And yet, most everyone would say this candidate is very electable! Why? Because if they got nominated AND remained a moderate, this characteristic would serve them well in the general election.

          Hopefully this analysis and example are fully digestible without any reference to correlation v. causation, and without wading into how betting markets actually work.

        • Daniel:

          Agree. New information is revealed all the time by all sorts of things, so I expect prices to change constantly. Even if no information is revealed by events you think should matter, prices will change because market participants do random things like change their minds for no good reason, sell for liquidity purposes, or start taking a nighttime cold medicine that makes them do wacky things.

        • Terry, the factors you mention are reasons for prices to fluctuate through diffusion through time. What I meant was if they announce the nominee on monday morning 1 minute before open of market and it’s based on the close of market prices on Friday, there would be a price shock at opening on monday, because the information that the candidate has been nominated is information that makes some of the scenarios incorporated into the previous p(win | nominated) pricing have zero probability, so it drops out of the mix, and other scenarios that were deemed relatively unlikely are now given much higher probability… the weighted average across all the scenarios changes instantaneously when that announcement is made.

          This is not just a diffusive jiggling caused by a few people taking cold medicine or needing to exit their positions or getting new infusions of capital they need to invest in their funds or whatever.

          This kind of thing happens for example when the Fed comes along and announces rate target changes… The market has priced in the average effect over all the possible rate changes they might make, but when one particular one is realized, and particularly when it’s one that was deemed unlikely by the market, it causes re-pricing throughout the market in a shock. This is why the Fed likes to hint at what its likely to do ahead of time so that prices equilibriate more slowly and people aren’t caught off guard.

          If the Democratic party nominated a person based on pricing, and that candidate had relatively high p(nomination), the size of the shock on p(win|nominated) would be smaller. When the p(nomination) was small to begin with, the shock on p(win | nominated) would be potentially dramatic.

          Ram: I think we’ve come to the heart of the matter. what is the definition you want to use of “electability”? it seems you want basically “how electable is the candidate if the candidate has the properties the candidate has now, but no longer has to vie for nomination”, whereas Andrew seems to want “what is the probability that the candidate who is nominated having the properties we can infer about that candidate at the later time of nomination, also then goes on changing their properties and eventually the election is held and the candidate wins”

          Neither one seems realistically what we want to know. Really what we want is p(elected | we hand the nomination to the candidate now), which incorporates the ways the candidate would change through time between now and election, but not the effect of the nomination process on those changes… essentially like an ordinary differential equation which has two kinds of forcing: nomination campaigning, and election campaigning… and we eliminate the nomination campaigning

        • Carlos:

          I see what you are getting at now. Your Canada-war example is very good.

          Candidate B is more “electable” conditional on a war with Canada, but less “electable” conditional on no war with Canada. The problem is that Pr(Win|Nominated) measures only the first conditional probability. Our best estimate of “electability” today is a weighted average of the two conditional probabilities.

          I have probably just muddied the waters. I do that a lot.

        • Daniel,

          I’m deliberately trying to avoid talk of intervention since this seems to be confusing the discussion, but I think you and I have the same concept of electability. As you can see in my comment, I’m not necessarily talking about the characteristics they have today. I’m talking about our forecast of how their characteristics would evolve over the course of the election, were they not required to evolve in any particular ways to secure the nomination first. Andrew’s concept bakes into this forecast that things have already evolved in ways that make them the nominee. But when deciding who to nominate (whether as a voter, a donor, an activist, or whatever), we remove from consideration the factors that influence nomination itself, since we’re substituting our judgment for these factors.

        • Carlos

          let me re-phrase that:

          “Obviously becoming more nominate-able at t+1, increases overall electability, on average.”

          I was thinking of the political & campaign skills, polish, support, etc. Clearly there are edge cases that we can cook up.

        • Carlos

          Ps. I was responding to your message way above, where you wrote:

          > Obviously becoming more nominate-able at t+1, increases overall electability

          Not necessarily. One can easily imagine a scandal affecting the party that changes the nomination probabilities and at the same time reduces the probability of winning the election for all of them. (By the way, I was joking about the “contrivedness” of the examples.)

        • Carlos writes:

          The disagreement, if any, was about the following claim:

          « the probability that a candidate wins given that we *observe* them winning the nomination is different from the probability that a candidate wins given that we *intervene* so that they win the nomination. »

          I think we all agree on most things. I don’t think anyone ever disagreed on this.

          Ram writes:

          What we mean by “electability” is how a candidate who has the characteristics she *actually has* would perform in the general election. What we do NOT mean by “electability” is how a candidate who has characteristics she *is predicted to have given that she was nominated* would perform in the general election.

          I had a feeling everything would hinge on our definition of electability, and our model of the election process.

          A moderate candidate that is trainable may not currently have characteristics A that serve one well both in the primary and in the general, characteristics that a more polished extreme left candidate currently has. These characteristics can be developed, especially if one has what it takes to get through a primary. Betting markets *can* price in these characteristics now, with bets on nomination, but markets may also price in potentially fixed characteristics B that speak to electability in the general but mean nothing in the primary (like being moderate), with bets on winning, or characteristics C that mean something for both (like charisma), with bets on both. The estimate of some of these characteristics B, and their relative importance in the general, may not change at all conditioning on being nominated. Sure the estimate of characteristics A may change, but that is fine, making it through the nomination develops the candidate on certain dimensions, and also mechanically gives an electability bump by being awarded the party label. Of course, there is the caveat that you note with your nice example of the moderate moving left. Some of these characteristics B may not stay fixed if the candidate needs to get through the primary. This is a path we have to average over when thinking about what P(Win| Nominated) means.

          The question becomes, what do we want electability to mean? I’d want to ignore certain actual characteristics that I believe will change if they make it through the nomination process. This is what I was getting after in my .9 vs .1 example, which I presume you don’t have an issue with. The others, like having to move left, I think relate to general electability also because we have to be realistic about what a presidential candidacy would like if this person wants to represent the Democratic party. For example, Bloomberg’s electability as a Democrat and his electability as a Republican are different things. I don’t see why we would want to talk about Bloomberg’s electability as some general 1-D construct. I think it is a contingent thing, as I have mentioned in other comments.

          In this sense, my view of electability also does not gel with Daniel’s here:

          Really what we want is p(elected | we hand the nomination to the candidate now), which incorporates the ways the candidate would change through time between now and election, but not the effect of the nomination process on those changes…

        • Joshua said:

          Carlos writes:

          The disagreement, if any, was about the following claim:

          « the probability that a candidate wins given that we *observe* them winning the nomination is different from the probability that a candidate wins given that we *intervene* so that they win the nomination. »

          I think we all agree on most things. I don’t think anyone ever disagreed on this.

          Well this is discouraging. We don’t even agree on what we disagree on. Hello rabbit-hole.

        • Ram,
          Hypothetically and pedantically it might be conceptually fine to ask P(Win | do(Nominated)) if and probably only if the DNC was governed by a dictator and he or she could fabricate the primary election secretly and arbitrarily — and all outsiders have no idea and still bet in the same way as if the dictator does not exist. In this case, the betting market are dealing with a different probability P(Win | Nominated) than the dictator P(Win | do(Nominated)).
          On the other hand even in this case we could also say the betting market fails to calibrate the probability by ignoring whether the dictator conduct that do operator or not. A sophisticated betting market should provide a calibrated calculation: P(Win | Nominated)= P(Win | dictator , Nominated)P(dictator)+ P(Win | non-dictator, Nominated)(1-P(dictator)), aka MRP.
          That being said, asking the causal effect of “X wining primary” is almost as ambiguous and impractical as asking the causal effect of gender or age.

        • so this is a good way to think about the issue… after the dictator is revealed, p(dictator)=1 and P(win|nominated) becomes p(win|dictator,nominated) which may be much smaller than it was before (or larger… in general much different)

        • Last month I was discussing this very point, but instead it was about Trump’s probability of being convicted in the Senate, conditional on being impeached by the House.

          With the usual caveats, betting markets allow us to calculate the implied Pr(Convicted by Senate | Impeached by House). These are expected *future* beliefs about Trumps convict-ability. It is a statement about his *average* future convict-ability across a range of scenarios that lead the house to impeach Trump. This has nothing to do his current convict-ability given our knowledge of where the Senate is at.

          But Trump’s current convict-ability in the Senate is irrelevant as defined by an intervention doesn’t exist (just as Yang’s current elect-ability doesn’t exist). The Senate is not going to vote on conviction/acquittal without the House voting to impeach. Itai Sher and I had this discussion here. This is a version of Yuling’s point, as I understand it. (I love the “causal effect of gender” analogy).

          Returning to nominations/elections and Ram’s & Carlos’ point: the only way Yang gets the nomination is after a long primary process, i.e. it is only Yang’s elect-ability in Nov 2020 that matters, not Yang’s Nov 2019 elect-ability based on an intervention, which doesn’t mean anything (of course is general political ability matters, but that is a different object). Outside of a hand-of-God intervention, any other feasible intervention that gets Yang nominated will involve him garnering enough popular support to beat the competition. This means that Pr(Convicted by Senate | Impeached by House) is the version of elect-ability we are after, not Pr(Win | do(Nominated)). While I do think P(Win | Nominated) != P(Win | do(Nominated)) is an interesting distinction to think about and helps clarify what aspects of our current beliefs aren’t relevant (e.g. in the Trump example), in this case it doesn’t exist and isn’t relevant for 2020 elect-ability (as Andrew mentioned in a later comment).

        • *TYPO*

          after my comment above is released from moderation (due to links) the first sentence of the 3rd paragraph should read:

          “But Trump’s current convict-ability in the Senate as defined by an intervention doesn’t exist (just as Yang’s current elect-ability doesn’t exist)….”

        • Joshua, this is one reason I don’t like the do calculus approach… there is often no one thing we can call do(X), there may be tens or hundreds of ways to make X happen, and they don’t all have the same effect.

          When calculating a Bayesian probability, we base it on a knowledge base. The probability of Trump to be convicted at a future date if he is impeached now is based on all the different ways it could happen… further whistleblowers could show up, audio recordings could be produced, maybe Trump could assault someone, a criminal conspiracy between Trump and certain senators could be discovered…. we give each of these some component in the probability…

          on the other hand… later down the road it will turn out that some things did happen and others didn’t… and the probability at that point will be different because it will assess a different set of possibilities.

          the stuff about “intervention” in this case seems to me to just be another instance of this. If someone comes along and changes Yang’s standings by advocating strongly for him to Party insiders, or someone releases a bunch of incriminating video of Trump plotting to have Pelosi killed… or whatever… the knowledge base we have changes a lot and as a result the probabilities change a lot.

        • Perhaps this will clarify what I’m talking about for those less familiar with do-calculus notation:

          Given the true probability measure P0:

          [1] P0(Y = y[i] | X = x[j]) = Sum_k P0(Y = y[i] | X = x[j], Z = z[k]) * P0(Z = z[k] | X = x[j]) for all i, j

          Now suppose we intervene on this system, fixing Z = z[1]. This induces a new probability measure P1, of which the following holds for all i, j:

          [2] P1(Y = y[i] | X = x[j]) = Sum_k P1(Y = y[i] | X = x[j], Z = z[k]) * P1(Z = z[k] | X = x[j])
          [3] P1(Y = y[i] | X = x[j], Z = z[1]) = P0(Y = y[i] | X = x[j], Z = z[1])
          [4] P1(Z = z[1] | X = x[j]) = 1
          [5] P1(Z = z[k] | X = x[j]) = 0 for all k != 1

          As a result:

          [6] P1(Y = y[i] | X = x[j]) = P0(Y = y[i] | X = x[j], Z = z[1]) for all i, j

          In general, then:

          [7] P1(Y = y[i] | X = x[j]) != P0(Y = y[i] | X = x[j]) for all i, j

          This is the sense in which intervening on a system can change the probabilities that describe that system.

          Absent any such intervention, the governing probability measure is P0, and we therefore would expect an efficient binary option market for candidate A winning the election to obtain a price of P0(Candidate A Wins) in equilibrium, and a similar market for candidate A being nominated to obtain a price of P0(Candidate A Nominated) in equilibrium (for simplicity, I’m ignoring pricing kernel issues). If we assume

          [8] P0(Candidate A Nominated | Candidate A Wins) = 1

          then the ratio of these prices gives us a good estimate of P0(Candidate A Wins | Candidate A Nominated) by Bayes’ theorem. Mankiw is defining this quantity as electability, and assuming the market is adequately efficient. Gelman agrees with this definition, but has doubts about the efficiency of the relevant markets.

          My contention is that, market inefficiency aside, what we care about when we talk about electability is some version of P1(Candidate A Wins | Candidate A Nominated). Given [7], even in an efficient prediction market we wouldn’t expect Mankiw’s calculation to be estimating the right quantity.

          Why do I think this? If the only scenarios (the only values of Z) in which a candidate is nominated are those in which the candidate wins, then by the Mankiw/Gelman measure the candidate is maximally electable. But there may only be one such scenario, and that scenario may be something like an asteroid striking a debate featuring all of the other contenders. It seems obvious to me that in this case, P0(Candidate A Wins | Candidate A Nominated) is not telling us anything about electability. Indeed, the reason the asteroid is the only scenario in which the candidate is nominated may be that everyone knows this candidate is maximally unelectable!

          I agree with Yuling and Josh and others who think that it isn’t so clear how to specify the relevant intervention here (in terms of what fixing Z = z[1] means), but that difficulty doesn’t mean that the pre-intervention conditional is a good measure of electability. Understanding that what we’re after is not a pre-intervention average of scenarios under which the candidate might be nominated, but instead some (admittedly opaque) post-intervention quantity, is useful in guarding against naive interpretation of P0(Candidate A Wins | Candidate A Nominated), even if it doesn’t direct us to a superior alternative.

        • > the only way Yang gets the nomination is after a long primary process, i.e. it is only Yang’s elect-ability in Nov 2020 that matters, not Yang’s Nov 2019 elect-ability based on an intervention

          The question discussed by Ram is to what extent that particular definition of electability does matter and is properly understood and in particular whether it is a reason to support a candidate.

          For a very simple example (the question is about principles and not specifics anyway), imagine two candidates A and B. One will be nominated, by a small subset of voters, to face Z in a general election.

          A and Z are very similar and widely liked and if the compete they have 50% chance. B is quite extremist and nobody really likes him. He doesn’t stand a chance unlike there is a new war against Canada. Most people would vote for him in that case.

          The nominated candidate will be A unless there is a war because in that case B will nominated. Electability is 50% for A and 100% for B. But who cares? It’s definitely not a reason to nominate B rather than A in a no-war scenario.

          > any (…) feasible intervention that gets Yang nominated will involve him garnering enough popular support to beat the competition.

          I had the impression that the reason why electability is discussed in the first place is the divergence between “support that gets someone nominated” and “general support”.

          And whether such an intervention is possible or not is irrelevant to the point that people may misinterpret P(A win|A nominated) to be the same as P(people prefer A to Z) when it’s not necessarily so.

        • +1

          Anyone still mystified by my comments can just read Carlos’s comments and decide whether they agree with Andrew that the concept of electability is “exactly” P(Win | Nominated).

        • +1 to Carlos’s comment above.

          The example Carlos gives of candidates A, B, and Z explains the situation excellently without requiring understanding technical aspects like Pearl’s do() notation, potential outcomes notation, or issues of whether the intervention is well defined. A great clear example.

          If anyone reading this thinks Ram is wrong and Andrew correct, please read Carlos’s comment. You can search for the phrase “new war against Canada” if you are unsure what comment I am referring to.

        • Frankly I also don’t understand where the disagreement with Ram/Carlos is coming from, given that in so many words, what they both seem to be saying is that an observed correlation between Yang being the nominee and Yang winning the general does not imply that causing Yang to win will tell us something useful about his odds of winning the general… am I missing something here?

        • The situation is particularly of interest when the probability of nomination is small. That essentially means that there are only very unusual circumstances where the nomination would occur. That’s very different than a more typical candidate. The fringe candidate should have a high p(win|nominated) specifically because the only likely way to get the candidate nominated is if things turn out very strange in that candidates favor, like Carlos’ war on Canada example.

          On the other hand for a more standard candidate, such as an existing senator or congressman or governor, the world doesn’t have to be particularly odd to get the nomination, and so the p(win|nominated) is an assessment of a world that isn’t unusual, one more like what we think of today’s world. hence, the post nomination assessment of p(win|nominated) should be similar to the pre nomination one.

        • Hi Ram-

          This conversation is interesting.

          On your comment here here, I think everyone began in agreement that intervening in a system changes the probability, and that the betting market conditional probability refers to future contingencies. The point of my Trump example was to show that reasoning about Pr(Y|do(X)) leads us to being thinking about Trump’s support in the Senate if the trial were held today, just as it leads us to being thinking about how Yang matches up against Trump among the general public right now. My view is that this thinking is a mistake, and that we want to be thinking about Pr(Y|X), given *what we know* about these specific environments. (Also, perhaps it is on me, but I fail to see what your reference to the do-calculus, or your derivation, adds to the issues we are focusing on)

          The disagreement appears to lie on whether electability, however we want to define it, has anything to do with the intervention-related probability. My understanding is that you (and perhaps Carlos) believe that the intervention probability is meaningful. You write: “what we care about when we talk about electability is some version of P1(Candidate A Wins | Candidate A Nominated).” I don’t think this hypothetical construct means anything even in theory, for the reasons described above. Daniel Lakeland has some further nice points on this.

          You write that when it comes to electability, “in an efficient prediction market we wouldn’t expect Mankiw’s calculation to be estimating the right quantity.” I don’t think Mankiw (or Andrew) claimed that it does. Mankiw wrote that it gives a “clue.” This is better than what the intervention-related quantity can give, which is either (i) a fantasy-land where nothing & everything is true (because an intervention does not exist), or (ii) a world in which elections and elect-ability no longer have any relevance (because the intervention reveals a world that operates according to entirely different laws).

          By contrast, under the (heroic) assumption that this is a prediction market operating as promised, which is Mankiw’s (and Andrew’s) background assumption, then Pr(Win | Nominated) is closer to a meaningful notion of electability than Pr(Win|do(Nominated)). In this case Pr(Win | Nominated) incorporates the beliefs of those with superior info who are willing to bet on it. Importantly, we are talking about a real election process and not an artificial example designed to make Pr(Win | Nominated) unrelated to intuitive notions of electability by removing relevant structure (e.g. in Carlos’s simple example he considers a single contingency, rather than averaging across contingencies).

          In contrast to contrived examples, a real election has candidates with qualities that don’t play well in the primaries but do play well in the general. In a real election a candidate can go from low name recognition (and low “general support” as Carlos writes) at the beginning of a primary, to high name recognition at the end. In a real election the primary process is tough; getting through it and winning can be transformative for a candidate, especially for a moderate candidate with little experience or campaign infrastructure. Pr(Win | Nominated) can give a us clue; in fact, I agree with the betting markets here—most moderate or non-polarizing candidates should have better conditional odds in the general than Warren right now—winning a marathon nomination with a competitive field would involve filling in many gaps for an inexperienced candidate without name recognition, campaign infrastructure, but who has qualities that may match-up well in a general. It would be transformative. Pr(Win | Nominated) suddenly becomes relevant in this setting for sussing out any additional private info on whether the candidate has qualities that could play well in the general if the gaps are filled. Imagine a completely uninformed financier who is trying to decide whom to support in the primary, and whom to not support. In a betting market that functions as intended, a completely uninformed financier who wants Trump to lose to anybody with a pulse should pay attention to Pr(Win | Nominated), at the very least it reveals that the candidate may play better in the general according to the median voter theorem. If you accept this, then surely you accept that Pr(Win | Nominated) relates to electability.

          Ps. As Andrew mentioned, there are many definitions of electability, and we could easily get side-tracked into discussions on what that means. I don’t think electability exists as an attribute of a candidate, i.e. there is no unmeasured structural parameter we are trying to get after. That would be like saying that the horse Seabiscuit has some characteristic called “winnability.” Sounds pretty silly. Instead, like horses, candidates have multiple-attributes, and they all feed into determining their chances given a particular line-up.

          PPs. On Daniels’s point, In the context of an election, I don’t think we should view a low probability event as something very strange happening, like a single intervening Black Swan event that wipes out the competition or vaults a candidate to the nomination without any primary process. Rather think of it as a sequence of events, actions, and efforts of the candidates that shape the candidate and the competition and leads the low probability candidate to win. Right now betting markets say that someone not Biden, Buttigieg, Warren or Sanders has a better chance winning the nomination when compared to any *single* one of those four, i.e. current front runner is Not-Biden/Buttigieg/Warren/Sanders. Conditional on that happening, Bloomberg has a decent shot to be the candidate, but it is more likely the candidate won’t be him.

        • Josh,

          Thanks for your comment. I’ll try to respond to each paragraph.

          (1) Not sure I understand the temporal dimension you’re bringing into this–could you clarify? And fair enough about the formalism. Andrew repeatedly implied I was making elementary probability errors, and I was trying to help him point to the specific error I made. But I agree that it isn’t essential to this discussion, as Carlos’s notation-free examples illustrate.

          (2) I think we agree that P(Candidate X Wins | Candidate X Nominated) is a weighted average of the random variable P(Candidate X Wins | Candidate X Nominated via Causal Process Z), where the weight is P(Candidate X Nominated via Causal Process Z | Candidate X Nominated). Consider a particular causal process–namely, an asteroid striking a debate featuring all other candidates. If this scenario obtained, Candidate X would be nominated, and would probably win (e.g., due to national sympathy for the party). I think we agree, however, that this fact has no bearing whatsoever on Candidate X’s electability. (Imagine bringing up this scenario in a conversation about electability!) And yet, if Candidate X is very unlikely to be nominated, this scenario may receive a decent amount of weight in the weighted average mentioned above, simply because there aren’t many other scenarios in which Candidate X is nominated. At a minimum, doesn’t this mean that P(Candidate X Wins | Candidate X Nominated) is a poor measure of electability for long shot candidates?

          (3) OK, maybe Mankiw has a more subtle view in which P(Candidate X Wins | Candidate X Nominated) is merely a clue about electability, but is not electability as such. I agree with this, as correlation is often a clue about causation. But I would not say, as Andrew did, that the concept of electability is “exactly” P(Candidate X Wins | Candidate X Nominated), just as I would not say correlation is “exactly” causation.

          (4) I hope that you don’t find my example in (2) contrived as well, but as I pointed out, P(Candidate X Wins | Candidate X Nominated) is a weighted average of more specific quantities, and I worry about some of these quantities being irrelevant to electability, yet receiving nontrivial weight. If you want to take do(.) notation out of the story, this is the way to do it. I think that the quantity that does the most justice to what people seem to care about when they talk about electability *is* a weighted average, but I think P(Candidate X Wins | Candidate X Nominated) *uses the wrong weights*. Carlos’s and my examples can be interpreted in this way, and it is because the weights are wrong that I think this conditional does not “exactly” capture electability.

          (5) As I said, I think P(Candidate X Wins | Candidate X Nominated) is often related to electability, just like correlation is often related to causation. We have two weighted averages of the same underlying quantities, but two different weighting schemes. This means they will often be related, but will sometimes be very unrelated.

          (6) Happy to avoid a semantic dispute. The relevant feature of electability I’m appealing to is that people use it to decide who to vote for, not to predict who will win the election. That’s why causality enters the story: acting versus observing.

          (7) Yes, this is what I meant by “Causal Process Z”. The examples I use are simply for comprehension, but the actual scenarios in question could be arbitrarily complex and multistaged.

        • Let me clarify my position using your contrived example of a financier who doesn’t care about who wins the election as long as it’s not Trump. The task is to find the candidate X (from the set {A, B, C, …} of eligible people who are not Trump) to support in order to maximize

          P(Trump loses | the financier supports X) = P(A wins | the financier supports X) + P(B wins | the financier supports X) + … =

          P(A nominated | the financier supports X) * P(A wins | A is nominated, the financier supports X) + P(B nominated | the financier supports X) * P(B is nominated | the financier supports X) + …

          I won’t dispute that Pr(A Wins | A Nominated) relates to electability (whatever it means). But you cannot just say that the terms in that calculation are the probabilities before the financier’s intervention (assuming for the sake of the discussion that they are precisely known).

          > You [Ram, not me] write: “what we care about when we talk about electability is some version of P1(Candidate A Wins | Candidate A Nominated).” I don’t think this hypothetical construct means anything even in theory, for the reasons described above.

          [6] P1(Y = y[i] | X = x[j]) = P0(Y = y[i] | X = x[j], Z = z[1]) for all i, j

          I think that in this case Z=z[1] corresponds to “the financier supports candidate X”.

          Apparently we all agree that probabilities will change when we intervene on the process (rather than just observing the outcome). The reason why we look at the implied probability Pr(Win | Nominated) and call it “electability” is to guide our intervention, so we cannot take it at face value. I think that was Ram’s main point. We could spend days discussing about the divergence between the number we would like to know and the one we can readily calculate. I think it will be quite case-specific.

          As a rule of thumb, I would expect that naive definition of electability to be less relevant when a larger intervention would be required, i.e. when the starting probabilities are small. One may also expect those probabilities to be relatively more noisy (I think I agree with Andrew there) and that noise will be amplified when we calculate the implied conditional probability. In summary, we have a badly estimated number that would be barely meaningful even if we knew it exactly.

        • Ram, I agree with your analysis, particularly your (2). The weights that are relevant to a longshot candidate are weights that consider situations that are unlikely to hapen (ie. there is low p(Nominated) for the candidate, so most things that might happen aren’t considered in the weights).

          I think your do() concept basically says “suppose we just force the nomination of this candidate” in that instance, the world is a very different place than what we were previously considering for p(elected | nominated). Now we must consider p(elected | nominated in this specific way) which is potentially a completely different number than the previous p(elected | nominated in any way at all)

          We might expect someone with a p(elected | nominated) ~ 1 but p(nominated) ~ 0 would have p(elected | nominated) drop to near 0 after being nominated through some means other than what was considered likely in the previous p(nominated)

        • Hi Ram

          (1) What I mean about the time dimension is that if people think about Pr(Convict|do(Impeach)), this think about intervening and impeaching right now, so they evaluate the current state of the Senate. Pr(Convict|Impeach) accounts for the impeachment process, what will be revealed in the future, and private info that you don’t have access to. It is the future state we are thinking about.

          (2) I don’t think P(X Wins | X Nominated) is a poor measure. I think you picked a particular process Z such that that has a low probability, and satisfies P(Win|Nominated, Z)=Pr(Nominated|Z)=1. I would argue the processes Z that P(X Wins | X Nominated) averages over include many more in the category such P(Win|Nominated,Z)≈P(Win|Nominated) but P(Nominated|Z) > > P(Nominated). Please see my comment “November 28, 2019 at 1:24 am” for more.

          (3) I don’t understand what you are getting at with correlation vs. causation. The market probability P(X Wins | X Nominated) can have very much to do with electability, and depending on how you model things, it can be “exactly” electability.

          (4) I do find your asteroid example contrived, but not in a pejorative sense. I only meant it by definition: it is a pathological and unnatural example that you constructed to illustrate your point. I think taking it as one contingency among many that are averaged over is less contrived than Carlos’s example which it is the only contingency. Of course, Carlos may have meant that he was focusing on one contingency among many and just didn’t mention that. Anyway, my argument is that the contingencies Z such that P(Win|Nominated, Z)=Pr(Nominated|Z)=1 are improbable. More common are the contingencies such that, for example, Pr(Nominated|Z) > P(Win|Nominated, Z) ≈ P(Win|Nominated)> P(Nominated).

          (5) I am not sure what you are communicating beyond 1-4.

          (6) This may be the key to everything. We haven’t focused in on what we mean be electability. You are assuming that people are voting in the primary and, at least in part, look to betting markets to guide them on whom is most competitive in the general. That is different than the example (i) I detailed in comment “November 28, 2019 at 1:24 am.” I think we can still handle this version of electability, but we need to assume the market is in equilibrium at time t awaiting time t+1 information shocks (e.g. will a financier enter with $ at time t+1, look at P(Win|Nominated), and work on making X more nominate-able?). At time t, P(Win|Nominated) can still look like electability, depending on your model of the world.

          (7) Okay, good, and these are less contrived and have more realistic relationships between conditional probabilities than the simple ones.

          Ps. Carlos:

          (8) the financier can operate in any example, whether contrived, or more realistic (note: I did not mean contrived in a pejorative way; still, I should have used a different word).

          (9) You write “…you cannot just say that the terms in that calculation are the probabilities before the financier’s intervention (assuming for the sake of the discussion that they are precisely known).” Not sure what you mean, but I do claim that the term P_t(Win|Nominated) in the betting market, e.g. in example (6), is a probability before the financier (potentially) emerges and intervenes at time t+1—all future contingencies are baked-in.

          (10) You write: “The reason why we look at the implied probability Pr(Win | Nominated) and call it “electability” is to guide our intervention, so we cannot take it at face value. I think that was Ram’s main point.” Yes, I think we hit that with Ram in (6) just above. In worlds in which most interventions Z satisfy P(Win|Nominated,Z) ≈ P(Win|Nominated) and P(Nominated|Z) > > P(Nominated), or something close, I think we can use P_t(Win|Nominated) to guide our interventions. The question depends on how we model our world.

          (11) You write: “As a rule of thumb, I would expect that naive definition of electability to be less relevant when a larger intervention would be required, i.e. when the starting probabilities are small. One may also expect those probabilities to be relatively more noisy…” We are ignoring noise, by assumption markets operate as intended and are liquid with smart money. In this case even small probabilities are meaningful. Also, the intervention doesn’t have to be that large, there is still time for candidates not in the top four, and while any given candidate not them has a small probabilities, the collective has a greater chance when compared to any single member of the top four.

  3. I tweeted out a similar estimate of electability of the candidates. But I did not compute this for candidates with only a few percent chance. I am curious what you think of my argument:

    If Yang has a 3% chance of winning the nomination, there is nearly no upside to buying a no for 97 cents and get one dollar after a year. It ties up a lot of your money, you could use to make other bets. A decade ago you would have been better off putting your money in a bank. The yes share only costs 3 cents and if lucky would be worth a dollar. That is a nice upside.

    So I was thinking that together with interest, there may be a tendency for odds at PredictIt to be closer to 50% than the real odds. That would also fit to all odds (being well below 50%) summed together being above 100%.

    Even without such a bias, there is an uncertainty in the odds and for a candidate that has 3% chance of winning the nomination and 2% of winning the election, this uncertainty will produce a huge uncertainty in the electability.

  4. I’ve noticed a lot of odd things like that on PredictIt markets. For example, now Yang is still surprisingly strong at 7%, Bloomberg is 11%, and Hillary Clinton is at 6% to get the Democratic nomination. I think it’s partly because most participants don’t have much at stake, and often use bets to declare their allegiance or look for a big payoff and bragging rights by betting on long shots.

    • ” I think it’s partly because most participants don’t have much at stake…”

      Actually I think the opposite is true: it’s the people betting on Sanders/Warren/Bidden that are betting above the risk. The convention and election are a political lifetime away. Early leaders have a long history of disappearing. The leading candidates have nearly a *year* of intense scrutiny to endure before the election and all the leading candidates are problematic in one way or another. Biden is among the leaders because there’s an uneasy sense that the broader field is too far left and unelectable.

      What you’re seeing is that it’s not clear yet what will happen but some people are – probably wisely – betting that there more shakeups are coming, and people like Bloomberg, Yang and Clinton may be in a strong position to take advantage of them. Bloomberg and Clinton especially, because they have substantial national name recognition and both are more centrist than loony left.

  5. Yang’s a fringe candidate, right? He’s polling at 0.8% on Real Clear Politics right now. I went over to Predictit and it says you can buy a Yang contract for the Democratic nomination for the price of 9%. Now, OK, sure, at 0.8% in the polls there’s room for improvement. But 9%??? Seems like a lot.

    Not surprising at all. The two numbers are very different things. The 0.8% is vote share if the election were held now. The 9% is the probability of being nominated almost a year from now. No reason they should be equal.

    The key intuition here is volatility. A lot will happen in the next year, and there is a non-trivial probability that things will break Yang’s way. The 9% is the probability something will happen to shake up the current situation in Yang’s favor — which is very different from what the 0.8% measures.

    Take an extreme example: two candidates, Bozo-T-Clown and George Washington. The two are polling at 0.8% and 99.2% respectively. But, there is a 90% chance George will die before the convention (or that he is already dead, he is after all, 322 years old in this scenario). What is the probability Bozo will be the nominee? 90% >> 0.8%.

    Caveat: yes, there is some market inefficiency inflating Yang’s price because longshots tend to have inflated prices (think out-of-the-money options).

    There are also reasons for Yang’s price to be higher than for most longshots. A bet on Yang is a bet on a specific scenario where things break in favor of a cheerful and rational candidate. Yang has this space pretty much to himself. There is a lot of latent fear of the crazy candidates, and voters haven’t squarely confronted the downsides of the elderly candidates. (Remember that Reagan was seriously impaired by senility in the later years of his presidency.)

    Bloomberg and Hillary’s prices are what they are for similar reasons: there are scenarios where things break their way. Bloomberg is also running as non-crazy, but is elderly. Hillary is both elderly and running as kind of the you-want-crazy?-hold-my-beer candidate.

    Also, note that all the odds add up to > 100%. Currently, they add up to 128% (although 10% of this may be due to the fact that prices cannot be less than 1%). This gives some idea of how much inflation there might be for longshots.

    • This is pretty basic options theory.

      An option that would be worthless if exercised today (because the option is “out of the money”) will sell for a positive price because there is a chance that the stock price will rise in the future. The more volatile the stock price, the higher the option price.

    • Terry:

      Sure, I agree that a 9% chance for a candidate who is polling at 0.8% is possible. It just didn’t seem so reasonable in this particular example! After all, there were lots of candidates polling much better than Yang at the time, with no particular reason to believe many of them would die (to use your example) or drop out soon. 9% seemed like a big number, considering everything. But, as discussed in my above post, there were various reasons not to take this number seriously. What bothered me about Mankiw’s post was that he didn’t seem to be just taking this as an amusing numerical exercise. He really seemed to think that these numbers meant that we should consider Buttigieg as the most electable candidate. It’s funny because in other settings economists are very aware of data quality. Perhaps because the data came from a market, he forgot to be skeptical.

      • The value of the options approach is that it forces you to think explicitly about volatility due to future events (not the same as day-to-day volatility in prices). The heart of options theory and practice is volatility. Volatility is so central that “implied volatility” is routinely calculated and option price quotes are often given in terms of implied volatilities rather dollars (because underlying stock prices change continually).

        Right now, there are a LOT of fringe candidates with startlingly high prices, not just Yang. This tells me there is a lot of systemic volatility priced into everything.

        Joe 22
        Elizabeth 22
        Pete 21
        Bernie 16 (he just had a heart attack for Chrise sakes)
        Mike 13 (an elderly billionaire to run against the Orange Man?)
        Andrew 8
        Hillary 7 (say what?)
        Kamala 4

        I’m surprised Kamala is so low. I would think the black-woman factor would be more in play.

        Even Joe Biden is a bit of a fringe candidate at 22. The Hunter Biden stuff looks just awful … and the House is about to declare bullying Ukraine the worst thing on earth.

        My big takeaway is that Elizabeth is surprisingly weak at 22. The media is well-prepped to hype the woman-card to the skies, she doesn’t come across as dodderingly elderly, and the media will send any loony stuff she is saying now down the memory hole come the general.

        I would be very interested in a calculation of implied volatility. Defining it would be difficult though because these prices don’t behave like stock prices at all: too many large jumps and bounded by 0 and 1.

        I agree day-to-day prices are pretty noisy. But with a little time-series averaging and some humility that they are plus or minus 5% or so, and they are pretty useful. They are definitely saner than most political commentary. I go to them to get an idea of what is really going on … plus or minus.

  6. Interestingly, Yang in particular has gone from 0.8% polling to 2.8%, 6th in the race. That might be a bit of evidence that the betting market did see something useful, though 9% is still very high for a candidate with low single digit polls.

    • Peter:

      Yes, good point. Indeed, the fact that a candidate is ranked better in prediction markets than in polls should be an indicator that they are likely to improve in the polls. My argument re Yang should not be construed as that his then-ranking in the polls was the best predictor at the time. I was just saying that the implied 9% probability seemed too high.

      • “the fact that a candidate is ranked better in prediction markets than in polls should be an indicator that they are likely to improve in the polls. ”

        Doesn’t it work the other way around? ie markets take feedback from the poles? In effect that’s what’s happening in the stock market right? Corporate earnings and revenue are the “poles” – or the voice of the people. The people spoke when they bought iPods and iPads and left PCs and Window’s phones on the shelf. If the people don’t speak with their money, traders don’t buy the stock.

        Also the market isn’t instantaneous efficient and can give false signals. Let’s say some news indicates an impending disruption among the leadership. If that disruption knocks some leader out of the lead, then obviously all the other candidates are more likely to win, so they’ll all rise in the short term. Then gradually the market will sort out what the news means for each individual – in part by getting feedback from the poles.

        • the market, if it’s sufficiently active, should aggregate all the available information, both from polls as well as private polls, or other kids of research. if there is information out there which isn’t reflected yet in the polls it may be already in the market prices. if you see market prices dropping, you should expect polls in the future to drop, the assumption being that some other source of info is being used to price the market and it isn’t yet in the polls

        • Technically, the relation between prediction market probabilities and current polling is pretty tenuous because the prediction market probabilities are determined exclusively by what happens at the convention. The contract pays off if and only if the candidate is the nominee. Current polling is only tenuously related to this. Yes, stronger-polling candidates are more likely to be the nominee, but the long-shot candidates need something extraordinary to happen, and that something is variable for each candidate.

  7. A simple way to justify the Predictit numbers for the “fringe candidates” is the following (very crude) calculation:

    8 candidates
    4 fringe candidates
    A 50/50 chance that complete chaos will erupt
    If chaos does not erupt, one of the non-fringe candidates will win
    If chaos erupts, each of the 8 candidates are equally likely to win

    So the probability that a given fringe candidates will win = .5 x 0 + .5 x (1/8) = .0625

    And the probability that a given non-fringe candidate will win = .5 x (1/4) + .5 x (1/8) = .1875

  8. While I’m pleased to have changed some minds, I do worry that the big picture is getting lost. Here is yet another attempt to clarify said picture:

    There are candidate characteristics that directly influence not only whether a candidate is nominated by their party, but also whether they win the election. As such, if we *observe* that a candidate was nominated by their party, we can predict their characteristics to some extent, and thereby predict whether they will win the election to some extent. This predictability is embedded in P(Win | Nominated).

    If instead we *intervene* so as to nominate a candidate regardless of their characteristics, then we break this chain of predictability. By severing the link between candidate characteristics and whether the candidate is nominated, we thus change P(Win | Nominated).

    My claim is that when we speak of electability, we’re speaking of P(Win | Nominated) *after* such an intervention, not before. Why do I say this? Because when we assess whether a candidate is electable, we assess them in view of *the characteristics they actually have*, not the characteristics we would predict them to have, given that they somehow secured the nomination. It is *as if* we intervened in the way described, leaving their characteristics unchanged but delivering them the nomination nevertheless.

    For illustration, the favorability of a candidate in the eyes of the voting public is going to affect whether they win the election, but it is also going to affect whether they are nominated (too-unfavorable candidates will be passed over by primary voters due to concerns about their general election prospects). A low favorability candidate is not electable, because if nominated *despite* low favorability, this characteristic would harm her chances of winning.

    Observing that this candidate has been nominated tells us they must be adequately favorable, and this means they will not fare too badly in the general election. P(Win | Nominated) will thus be high. But in fact, this candidate *is not* adequately favorable, and it’s precisely for this reason that they’re not electable.

    Had we (hypothetically) intervened, and made the candidate the nominee despite low favorability, P(Win | Nominated) would no longer reflect the mistaken notion that the candidate is adequately favorable, and so would be appropriately low. It is in this sense that electability is about a post-intervention version of P(Win | Nominated), even though no actual intervention occurs or even needs to be contemplated to make sense of the idea.

    The key is what would happen in the general election if the candidate was nominated *regardless of their actual characteristics*. P(Win | Nominated) is measuring the wrong thing, because it is telling us what would happen in the general election if the candidate was nominated *due to having different characteristics than she actually has*.

    As I said I’m glad I’ve changed some minds, but a lot of the discussion has focused on long shot candidates, and other special cases. The larger point is that candidate characteristics confound the association between nomination and election, and electability is a question about the unconfounded association—that is, the causal effect.

  9. Andrew,

    The use of a slash / instead of a vertical bar | in writing a conditional probability seems to be preferred in the philosophical literature. I’ve read a number of philosophical papers on probability and inference and they invariably wrote P(A/B) rather than P(A|B). The vertical line seems to be universal amongst statisticians. They may be reading it “Probability of A on B” which would make sense.

  10. Terry,

    welcome to the confused crowd :-)

    Ram,

    I agree that your latest example where the candidate “moves” may be easier to understand than the case where the candidate is “fixed” but the environment is different in the “general” cases than it was in the “special” cases where it was expected to get the nomination. Most people has the intuition about a trade-off between being “extremist” enough to please those than vote in the primaries and being “centrist” enough to win the general election. Many people will also cynically deduce that the optimal strategy is to say something to get the nomination, something else to win the election and then do whatever you want.

    Joshua,

    I don’t think there has been much of a disagreement between you and the “anti-Andrew crowd”.

    I’m not sure about Andrew, honestly. In his last comment he said that “we just care about Pr(X,Y), where these probabilities average over all uncertainties”.

    Maybe I’m complicated things but I find P(X=winner, Y=nominated) to be complicated. It’s a marginal probability obtained integrating over all uncertainties the incredibly complicated joint distribution P(X, Y, Z).

    Those uncertainties evolve as things happen, the joint distribution changes over time, the marginal distribution changes over time. While most of those uncertainties are outside of our control, our actions also determine to some extent that the distribution goes in one or other direction.

    Some actions can have a big impact. Say I have one preferred candidate A and I have a video of the favorite for the nomination B in blackface or whatever. If I release the video I cause P(X,Y)=P(X|Y)P(Y) to change:

    – One can expect P(Y=A nominated) to go up as P(Y=B nominated) goes down.

    – It’s also reasonable to expect that in this case P(X=A wins) will increase. At least that was my reasoning when I decided to release the video!

    – P(X=A wins|Y=A nominated) could go in either direction (and there is no reason why it shouldn’t change). I don’t really care, my objective was to increase P(X=A wins). One reason why it could go down: the bar for A’s nomination is lowered so it may be now a worse nominated candidate that if it had managed to beat B without my intervention. One reason why it could go up: if there had been a nasty fight with B it could have been damaging for both regardless of who finally won the nomination.

    – P(“a candidate from my preferred party wins”) could also go in either direction. This may be a problem if my actual objective was to increase the party chances and the only reason I wanted to increase P(X=A wins) is because I noticed that P(X=A wins|Y=A nominated) was higher than P(X=B wins|Y=B nominated).

    • I’ll try to clarify.

      I wrote that I don’t think anyone disagreed on what Carlos wrote here

      The disagreement, if any, was about the following claim:

      « the probability that a candidate wins given that we *observe* them winning the nomination is different from the probability that a candidate wins given that we *intervene* so that they win the nomination. »

      My reading wasn’t that anyone was claiming that these were equal, just that this *intervene* thing didn’t exist.

      Of course I can’t speak for Andrew here.

      Andrew wrote:

      Electability” is just a word. It means different things to different people. As a statistician, I think “electability” is P(win|nominated), nothing more, nothing less.

      Ram Wrote:

      What I’m wondering is why anyone should care about P(Win | Nominated). My claim is that some people (e.g., Mankiw) care about this quantity because they think it is useful for identifying candidates who will fare better in the general election. My point is that this is fallacious.

      …later…

      (1) In general, P(Win | Nominated) != P(Win | do(Nominated))
      (2) When people are trying to figure out which candidates are more or less electable, what they’re usually after is some particular disambiguation of P(Win | do(Nominated))
      (3) Efficient prediction markets for who wins the election, and who wins the nomination, can be used to estimate P(Win) and P(Nominated)
      (4) P(Nominated | Win) = 1

      Ergo

      (5) Even ignoring issues of market inefficiency, we cannot in general estimate electability using ratios of these prediction market prices.

      This seemed to be the point of disagreement. Andrew disagreed with this. He seemed to acknowledge that he wasn’t trying to make his best case.

      I happened to be closer to Andrew’s view here, and I chimed in. I made an argument for why someone should care about P(Win | Nominated), and why it is not fallacious. This directly contrasts with the conclusion in (5). I have also made a case against premises (1)-(2). In particular, that (1) is ill-posed and (2) is not true.

      IMO the main point of disagreement has been discussed, it seems to boil down to what we want electability to mean, and how relevant we view those pathological examples to be for the actual election process.

      Ps. With regard to Carlos’ example just above with joint distributions, I am not sure what is being addressed that hasn’t been addressed already. I also think it would be dangerous to comment because we likely do not have common assumptions on our model of the electoral process, the nature of the betting market, and the allowable feedback between the two.

  11. Joshua,

    > this *intervene* thing didn’t exist

    I agree that there is not one single way to “intervene so that they win the nomination”. I have not even used the do() dotation in any of my messages and it doesn’t appear in that paragraph either. I think interventions exist. We can “not intervene” (a.k.a. “observe”) or we could “intervene” in different ways to increse the nomination probability of our preferred candidate. The resulting probabilty will depend on what we do (respect to the baseline where we do nothing).

    > [Andrew] “electability” is P(win|nominated), nothing more, nothing less

    Sure, and then the paragraph I quoted can be rewritten as follows:

    « the electability of a candidate given that we *observe* if they win the nomination is different from the electability of a candidate wins given that we *intervene* so that they win the nomination. »

    Ram’s first two points say simply (given that definition of electability) become:

    (1) In general, “electability if I don’t intervene” != “electability if I do intervene”

    (2) When people are trying to figure out which candidates are more or less electable, what they’re usually after is some particular “electability if I do intervene”

    I don’t think you need detailed models of things to say that in general “interventions” change the probabilities.

    The previous probability was a weigthed average over our potential actions, when we intervene one of those become real and the others vanish.

    What you may need a detailed model for is to justify that prediction markets prices give you good estimates of the “post-intervention” electabilities.

    Of course I agree that I’m not saying anything that we have not discussed already. Even Daniel agrees with me so it’s a good time to quit :-)

    • Sounds good.

      Ps. You wrote:

      > [Andrew] “electability” is P(win|nominated), nothing more, nothing less

      Sure, and then the paragraph I quoted can be rewritten as follows:

      « the electability of a candidate given that we *observe* if they win the nomination is different from the electability of a candidate wins given that we *intervene* so that they win the nomination. »

      I agree. I also think the “electability” of a candidate depends on which party they run for.

      My main point was not Ram’s premise, but the conclusion. I gave examples of how you could estimate with those ratios, and how the ordering would likely be informative.

      Anyway, good time to quit. Would be easier to hash out f2f.

  12. tl;dr
    It’s surprising to me that any of this is controversial. Here’s a simple, natural, numeric example where supporting the candidate with higher p(win | nominated) increases the probability of Trump winning. Therefore, it is incorrect to assume the candidate with higher p(win | nominated) is more electable.

    There is a ton of discussion and confusion in comments above, but the problem has a unambiguous answer: If your goal is to nominate the candidate who will be most likely to beat Trump, then there is no mathematical guarantee that you should support the candidate with the highest p(win | nominated) at the time of the primary. To see this, here is a simple numeric example in which supporting the candidate with the highest p(win | nominated) increases the probability of Trump winning the election.

    The example involves two Democrat candidates, A and B, and is shown by this conditional probability tree:

    https://i.postimg.cc/V6B6qYyM/Electability.jpg

    The first tree branch shows two possibilities: sexism being relatively less severe than polls suggest (P = 0.2), which advantages female candidate A, and sexism being relatively more severe (P = 0.8), which advantages male candidate B. The rest of the tree shows conditional probabilities of nomination and election to the presidency.

    Note that P(President = A | Nominated A) > P(President = B | Nominated = B), so that according to Andrew and Mankiw, we should view A as having greater electability.

    However, if during the primary we add delta in support of candidate A being the nominee (0.6 -> 0.6 + delta; 0.2 -> 0.2 + delta), then the probability of a Trump win changes to P(President = Trump) = 0.488 + 0.06 * delta. Counterintuitively, the probability of Trump winning increases when we support A!

    For full generality, we can also imagine that the strength of our support can differ if there is less sexism (0.6 -> 0.6 + delta1) vs. more sexism (0.2 -> 0.2 + delta2). Then P(President = Trump) = 0.488 + -0.06 * delta1 + 0.12 * delta2. So, to ensure our support for candidate A reduces the probability of Trump winning, we would need our support to be more than twice as strong in the world/branch with less severe sexism (delta 1 > 2 * delta2). It is not clear how that could be ensured, since we do not know which world we live in, and since the factor of 2 itself depends on the probability associated with each world, which may be unobserved by us.

    In summary, even in simple examples, there is no guarantee that the candidate with highest p(win | nominated) is the candidate to support if your goal is to beat Trump. Therefore I think it is wrong and misleading to call p(win | nominated) electability.

    Notes:
    * In Carlos’s nice example, a war against Canada is used instead of sexism. But I wanted to avoid using a future event for the split in the tree, to remove irrelevant considerations about how the probabilities will change.
    * My example assumes that changing the conditional probabilities of nomination by delta does not alter the other conditional probabilities in the tree. If you suppose delta is small perturbation (true for most voters!) then this is a natural assumption, and the issues with describing p(win | nominated) as electability still arise.

    • More:

      I agree that once you consider an intervention (even a small intervention such as the casting of a single vote), the probabilities will change. If the betting market has no vig and is efficient (an assumption that is clearly violated in this case, as discussed in my above post; indeed that was the main point of my post), it will reflect some consensus about the joint probability distribution right now, implicitly integrating over all possibilities of future interventions. Some aspects of the above comment thread remind me of discussions of Newcomb’s paradox.

      • Two points. (Sorry about the long response.)

        1 — Newcomb’s paradox:

        I agree that Newcomb’s paradox is fascinating and can be related to prediction markets. But the issues with p(win | nominated) as “electability” arise even in situations where Newcomb’s paradox does not apply. For example, don’t view the probabilities in the example tree I gave as values from prediction markets. Instead, imagine they are my subjective individual beliefs of what would happen if I didn’t vote in the primary. Then if I voted for the candidate with highest p(win | nominated), I would still be advantaging Trump!

        2 — Electability:

        As you’re likely aware, people run polls in which potential voters are asked to make a binary choice between Trump vs. Biden, or between Trump vs. Warren, or Trump vs. Sanders, etc. The results of these polls could be used to assign to each candidate a 538-style probability of winning against Trump if they were nominated, which I think would make a fairly good assessment of electability.

        However, the 1-vs-1 poll-based assessments measure a fundamentally distinct quantity from p(win | nominated). I’m not sure if you’ve observed this already or if it has escaped your notice.

        For instance, in the example probability tree I gave, P(President = A | nominated = A) was 54 percent. However, using the exact same numbers found in that probability tree, a 1-vs-1 poll-based assessment of the probability of A winning vs. Trump would give 44 percent in expectation (=0.2 * 0.8 + 0.8 * 0.35), including 80 percent if we live in the world with less sexism and 35 percent if we live in the world with more. In no case does the quantity equal p(win | nominated)!

        So even if there is no market inefficiency, even if polls are perfect, even if probabilities don’t change, and so on — even if the prediction markets and poll-based analyses in fact derive from the same numbers — even in this case p(win | nominated) is measuring a fundamentally different quantity from the 1-vs-1 comparisons that appear most naturally to be measures of electability.

        Given this, I think p(win | nominated) should not be called electability. It’s clear from Mankiw’s posts and others reactions (including, as far as I can tell Joshua Miller’s and your own) that p(win | nominated) is being viewed as a guide for electability with the same kind of information as 1-vs-1 poll-based results. But this is incorrect. Even in the best of cases where everything is based on the same underlying probability tree, p(win | nominated) is measuring a different quantity than 1-vs-1 comparisons.

        This raises a question: what do the 1-vs-1 poll-based comparisons measure, expressed in the language of probability? I *think* they measure P(President = A | do(nominated = A)), where the do() intervention is whatever each poll respondent vaguely expects would happen for A to be nominated. Actually, I think this would probably be a decent match, on average, to the kind of interventions that would in reality get the candidates nominated! So they are very relevant to electability.

    • More Anonymous:

      you wrote: “If your goal is to nominate the candidate who will be most likely to beat Trump, then there is no mathematical guarantee that you should support the candidate with the highest p(win | nominated) at the time of the primary.”

      I don’t think the mathematical guarantee was ever controversial, or that there was confusion about this.

      The disagreement was about claims that we shouldn’t care about P(win|nominate) if we care about electability, and that P(win| nominate by intervention) is what we are after.

      • Joshua —

        tl;dr: Thanks very much for your response. I think that p(win | nominated) will commonly be a poor measure of electability. In particular, I express a simple version of the problem in causal graphs, which show that unrealistic independencies are required for p(win | nominated) to be a good measure of electability.

        I agree with your point that we may want to care about p(win | nominated), which I take to be a fairly uncontroversial claim. However, I also think that p(win | nominated) will commonly be misleading when used as a measure of electability.

        Below, I show this by expressing a simple version of the problem in causal graphs. As seen in the graphs, p(win | nominated) will only be a good measure of electability to the extent that the characteristics that determine who wins the nomination are independent of the characteristics that determine who wins the presidential election. Since these characteristics are generally highly related and overlapping, I expect that p(win | nominated) generally provides a poor reflection of electability.

        For candidates A and B, one can also ask, “How often will p(win = A | nominated = A) > p(win = B | nominated = B), but B be more electable than A?” This occurs under the same conditions as the classic Simpson’s paradox, as can be seen in both the conditional probability tree example I gave before and the causal graphs below. In practice, I *think* that the conditions for Simpson’s reversal will be relatively common in the US primary system, since potential nominees who are more favored in the primary are often less favored for the presidental election (e.g., candidates far left or right of centre). However, I am far from sure of that.

        Here are the causal graphs:

        https://i.postimg.cc/YSLQHFcx/Electability-DAGs.jpg

        In Figure 1, the nomination result is determined by a set of background variables, while the presidential election result is then determined by both the background variables and the nomination result.* We would like to identify the candidate who, if supported to become the nominee, is most likely to defeat Trump. This candidate is the most electable.

        For simplicity, let’s start by considering interventions, do(Nomination = C), which establish candidate C as the nominee with certainty. Then, the most electable candidate has the highest P(Presidency = C | do(Nomination = C)). But as seen from the graph, P(Presidency = C | do(Nomination = C)) only equals P(Presidency = C | Nomination = C) if the variables which determine the presidency are independent of the variables which determine the nomination, which is unrealistic.

        As can also be seen from the graph, misinterpreting p(win | nominated) as electability is the same kind of error that people make when they describe an observed association between a treatment and a disease outcome as “the effect” of the treatment, without accounting for confounding — the same graph structure applies to both scenarios.

        A limitation to the analysis is that it does not account for the possibility that the choice of nominee changes the values of variables that determine the presidency. This is addressed in Figure 2. However, even in this more general case, unrealistic independencies between determinants of the nomination and the presidency are still required for P(Presidency = C | do(Nomination = C)) to equal P(Presidency = C | Nomination = C).

        Another limitation is that the intervention do(nominee = C) sets the nominee to candidate C with certainty, but most mechanisms of support for C that we are interested in will only increase the probability that the nominee is C, and the strength of support may itself depend on background variables. We can address this limitation by evaluating electability for stochastic and conditional interventions, but this does not remove the requirement for unrealistic independencies if you want P(Presidency = C | do(Nomination = C)) to be a good measure of electability.

        There are many complicated features of the real world that have not been addressed in the analyses above — dynamic changes in probabilities, vote splitting in the primary, and so on. But my point is that, since P(win | nominated) appears to be a poor measure of electability in the analyses that distill the situation down to its basics, we have no good reason to expect P(win | nominated) will become a good measure of electability if we add all the complexities on top.

        * For those unfamiliar with causal graphs: All nodes in the graphs are also determined by “disturbance” or “noise” variables that are independent with each other and with all other variables in the causal graph. By convention these are not shown, but are assumed present.

        • Dear More Anonymous-

          Thanks for taking the time to reply here. Perhaps it is on me, but I fail to see what the causal graphs could add here. I mean we have already noted the Simpson’s like structure, with nomination signaling info that increases general election odds. The point I have made in other comments is that all candidates share this. The unique characteristics that aren’t honed or brought out in the primaries, the ones that set the candidate apart, are what P(win|nominated) can capture, depending on how we model the election. In any event, we can’t test whose model is more right. Also: don’t worry, I wouldn’t use P(win|nominated) if I needed to make decisions, albeit for other reasons. :)

        • Thanks Joshua! In reply

          Perhaps it is on me, but I fail to see what the causal graphs could add here.

          The causal graphs show that, even in the most basic, stripped-down version of the problem, having p(win | nominated) be a good approximation of electability requires that the variables that determine the nominee are largely independent of the variables that determine the president. The requirement is implausible, so we shouldn’t expect p(win | nominated) to be a good approximation of electability.

          To my understanding, this conclusion addresses the heart of the discussions going on above.

          The causal graphs also show that observing “p(win = A | nominated = A) > p(win = B | nominated = B)” and concluding “candidate A has greater electability” is the same mistake as observing “p(healed | got drug A) > p(healed | got drug B)” and concluding “drug A has greater heal-ability.” Same graphs.

          I mean we have already noted the Simpson’s like structure

          I couldn’t find previous mentions of Simpson’s paradox, though maybe it was implied.

          In any event, we can’t test whose model is more right.

          I don’t intend the causal graphs as models really — they’re identifibility arguments. I’m using them to claim that p(win | nominated) identifies a fundamentally different quantity than electability. I think this point was made by Ram earlier too. To me, it means that our default position should be that P(win | nominated) is a poor measure of electability, and that substantial evidence should be needed to shake us from this default position. You seem to have been arguing the reverse.

        • Oops, I didn’t mean for my comment to be so nested. I’ll try again…

          Thanks Joshua! In reply

          Perhaps it is on me, but I fail to see what the causal graphs could add here.

          The causal graphs show that, even in the most basic, stripped-down version of the problem, having p(win | nominated) be a good approximation of electability requires that the variables that determine the nominee are largely independent of the variables that determine the president. The requirement is implausible, so we shouldn’t expect p(win | nominated) to be a good approximation of electability.

          To my understanding, this conclusion addresses the heart of the discussions going on above.

          The causal graphs also show that observing “p(win = A | nominated = A) > p(win = B | nominated = B)” and concluding “candidate A has greater electability” is the same mistake as observing “p(healed | got drug A) > p(healed | got drug B)” and concluding “drug A has greater heal-ability.” Same graphs.

          I mean we have already noted the Simpson’s like structure

          I couldn’t find previous mentions of Simpson’s paradox, though maybe it was implied.

          In any event, we can’t test whose model is more right.

          I don’t intend the causal graphs as models really — they’re identifibility arguments. I’m using them to claim that p(win | nominated) identifies a fundamentally different quantity than electability. I think this point was made by Ram earlier too. To me, it means that our default position should be that P(win | nominated) is a poor measure of electability, and that substantial evidence should be needed to shake us from this default position. You seem to have been arguing the reverse.

        • sorry for the delayed response, been away.

          yes, I am arguing the reverse, more or less, assuming prediction markets operate as promised.

          My previous arguments contained:

          1) Pr(good qualities for general| nominated)> Pr(good qualities for general), which is Simpson’s basically.

          2) Pr(good qualities for general, not good for nomination) can be incorporated into Pr(win|nominated) when *comparing* candidates, and it is more or less in practice. The point is a relative one. We should expect this in a prediction market that functions as promised, which is the domain of this discussion.

          In sum, my verbal arguments accounted for the causal graph feature that highlights variables determining nominee & president. I made this very distinction when discussing a candidate being charismatic (good for nominee & election) and being moderate (good for election). The qualities that are good for both get updated for all candidates (perhaps unequally, depending on the base—e.g. Yang more). Again, it is a relative point, and .2 vs .8 will mean something, as I mentioned before. Calibration is another matter, and it will depend greatly on assumptions.

    • That’s also a good example. Another one that everyone agrees with! Terry gave the best summary of this discussion: “We don’t even agree on what we disagree on.”

      Ram 23.11> the concept of electability is not captured by P(Win | Nominated) […] If my objective is instead *intervening* to hand the nomination to the most electable candidate, P(Win | Nominated) is not what I’m looking for.

      Andrew 23.11> You are wrong. The concept of electability is ____exactly____ captured by P(win|nominated).

      Joshua 28.11> Of course, we all agree — from the beginning, I presume — that we can construct simple models in which P(Win | Nominated) is a terrible measure of electability.

      Ram 23.11> Even ignoring issues of market inefficiency, we cannot ____in general____ estimate electability using ratios of these prediction market prices.

      Joshua 30.11> I gave examples of how you could estimate with those ratios, and how the ordering ____would likely____ be informative.

      Joshua 30.11> The disagreement was about claims that we shouldn’t care about P(win|nominate) if we care about electability, and that P(win| nominate by intervention) is what we are after.

      The disagreement seems to be that one side says “A!=B in general” and the other side responds “A~B sometimes”.

      “We care about X, not about Y” and “we care about Y to the extent that it may be close to X” are not contradictory claims either.

      • thanks for this Carlos. Multiple discussions was an issue, and I think that happened because the mathematical point was never acknowledged clearly. One caveat: Ram wasn’t just making the general mathematical point, he was also making the practical point, which is where disagreement arose. It wasn’t simply

      • Thanks for this Carlos. I also have been puzzled by the number of distinct conversations stemming from the original one. My focus has primarily been on the straight contradiction between your first quote from me and your quote from Andrew. And I agree with your concluding statement: I have no problem with the claim that P(Win | Nominated) carries some information about electability, and that under certain special conditions it may even give a good approximation of it. My original post was designed to remind people that these two concepts are distinct, and *not necessarily* good approximations of each other. I think (?) this has been broadly accepted, except by Andrew, who hasn’t retracted his repeated claim that I was making some sort of error (seemed to me the allegation was of an elementary probability confusion). I do think that his mention of Newcomb is interesting in this context, but the relationship between this case and that one is a discussion for another day.

        Thanks all for your feedback and insights, and thanks to Andrew for hosting this very interesting thread.

      • Carlos, thanks for the summary of people’s positions to date. Very helpful. I’ve tried to take your summary into account in my responses to Andrew and Joshua above.

        In short, I think P(win | nominate) is rarely a good approximation for electability, and that we should instead look to poll results that force potential voters into a binary choice between Trump and Biden, or between Trump and Warren, or between Trump and Sanders, etc. These polls target P(win | do(nominate)), and not P(win | nominate).

        • This is turned into a discussion of electability, which is getting away a bit, but I’ll bite. What is the evidence that head-to-head polls like these are predictive? For sure it won’t be if candidates differ greatly on name recognition.

          Regarding whether P(win | nominate) captures something meaningful, or is a good approximation, under our assumptions that betting markets work as promised, and elections operate as they do: none of our claim are testable, the assumptions don’t hold.

        • I agree that the head-to-head polls have problems like you mention. Thanks for raising that point. However, I’m saying that the head-to-head polls are targeting the right quantity (electability), while estimates of p(win | nominated) are targeting something else.

          Maybe thinking about treatments and patient outcomes could clarify:

          Suppose an unadjusted observational study found p(healed | got drug A) > p(healed | got drug B), while a fully-adjusted observational study found p(healed | do(got drug B)) > p(healed | do(got drug A)). In practice, it could be that the unadjusted study was a lot better quality than the adjusted study, but we shouldn’t let this mislead us into thinking that in general “p(healed | got the drug)” is a better measure of ability to heal than “p(healed | do(got the drug)).”

          Analogously, even if it were the case that the prediction markets have a lot better quality than the head-to-head polls, we still shouldn’t be mislead into thinking that p(win | nominated) is the better measure of electability.

  13. Since there are very basic disagreements about what we are talking about, my instinct is to build a model from the ground up being explicit about everything. Let me try that. Call the model “TerryLand”.

    There is a primary today in which we can vote. One month from today, there is another primary, the convention is two months from today, and the election is three months from today. Assume (at least for now) that our vote is not large enough to affect anything.

    At each point in time, other people’s votes are determined by a known stochastic model. There is a vector of state variables, z, that may affect anything. Note that calculations regarding “electability” are part of z since anyone can make those calculations. The candidate that gets the most votes in the two primaries wins the nomination. We do not make donations or do anything else that might affect anything … let’s keep this simple for now.

    As a first step, assume our only objective is to identify which candidate we hope will be nominated because he/she is most likely to win the general election. This is the clearest definition of “electability” I can think of. If you disagree, state the objective in plain English without using the word “electability”.

    Can we accomplish our objective using betting market information? (Assume the betting markets are perfect.)

    Next step: add an action we can take now that will affect the system. Our objective is to increase as much as possible the probability that a Democrat will win the election. How should we act? This is not a simple problem! You can’t just act to help the most “electable” candidate because different candidates may be affected differently by our action. (The action may affect the betting markets … assume we know how it will affect the betting markets.)

  14. Semantics aside – we all know that Sam Wang was not wrong when he ascribed a 99% implied probability to a Clinton victory – because he had ascribed a 1% probability to a Trump victory (less wrong ;)- the problem with these so called prediction markets is that, even when there is lots of liqudity, they are simply not Bayes Optimal. The bettng market on Brexit – see below – simply failed to factor in the polls – but yet was followed by FX TRADERS.

    These markets are driven by sentiment and fall into every known trap – confirmaton bas, motivated reasoning, wishful thnking etc …
    #
    https://www.bettingmarket.com/fools.html

    • Niall:

      Huh? Yes, Sam Wang was wrong on that assessment! The perverse incentive is that his overconfidence got him attention (see item 3 here). As I wrote: “After the election, Wang blamed the polls, which was wrong. . . . The mistake was not in the polls but in Wang’s naive interpretation of the polls which did not account for the possibility of systematic nonsampling errors shared by the mass of pollsters, even though evidence for such errors was in the historical record.”

  15. For what it’s worth, here is a fully mathematical statement of my understanding of the disagreement between myself and Andrew. When I have some time, I will put together an R script so folks can play with the model specification, and see how it affects each of our proposals.

    Notation:

    z: presently nonpublic information which influences whether candidate is nominated and whether candidate is elected
    z*: presently public information which influences whether candidate is nominated and whether candidate is elected
    u: presently nonpublic information which only influences whether candidate is nominated
    u*: presently public information which only influences whether candidate is nominated
    v: presently nonpublic information which only influences whether candidate is elected
    v*: presently public information which only influences whether candidate is elected

    x: 1 if candidate is nominated, 0 if not
    y: 1 if candidate is elected, 0 if not

    A, A*, B, B*, C, C*: unspecified marginal distributions
    f, g: unspecified functions

    Observational Model:

    z ~ A
    z* ~ A*
    u ~ B
    u* ~ B*
    v ~ C
    v* ~ C*

    x := g(z, z*, u, u*)
    y := f(x, z, z*, v, v*)

    Interventional Model:

    z ~ A
    z* ~ A*
    v ~ C
    v* ~ C*

    x := 1
    y := f(x, z, z*, v, v*)

    Andrew’s Proposal:

    Electability := P(y = 1 | x = 1, z*, u*, v*) in the observational model

    Ram’s Proposal:

    Electability := P(y = 1 | z*, v*) in the interventional model

    • Here is an R script. In the default specification, the probability of nomination is very low (a0 = -2), but the probability of election conditional on nomination is very high (b0 = 2). Moreover, there is unobserved confounding (a1 = b1 = 2). An example along these lines is Carlos’s war with Canada, where the candidate is very unlikely to be nominated unless we go to war with Canada, but if we do the candidate is very likely to be elected. Running the script with this specification results in (my understanding of) Andrew’s proposed electability score being very high, while my proposed score is very low.

      # install.packages(“xgboost”)
      library(xgboost)

      ### Specification ###

      # Number of simulations
      N <- 100000

      # Means of exogenous random variables
      mu.z <- 0
      mu.z.star <- 0
      mu.u <- 0
      mu.u.star <- 0
      mu.v <- 0
      mu.v.star <- 0

      # SDs of exogenous random variables
      sigma.z <- 1
      sigma.z.star <- 1
      sigma.u <- 1
      sigma.u.star <- 1
      sigma.v <- 1
      sigma.v.star <- 1

      # Coefficients for determining nomination status
      a0 <- -2
      a1 <- 2
      a2 <- 1
      a3 <- 1
      a4 <- 1

      # Coefficients for determining election status
      b0 <- 2
      b1 <- 2
      b2 <- 1
      b3 <- 1
      b4 <- 1

      ### Simulation ###

      # Generate exogenous random variables
      z <- rnorm(N, mu.z, sigma.z)
      z.star <- rnorm(N, mu.z.star, sigma.z.star)
      u <- rnorm(N, mu.u, sigma.u)
      u.star <- rnorm(N, mu.u.star, sigma.u.star)
      v <- rnorm(N, mu.v, sigma.v)
      v.star <- rnorm(N, mu.v.star, sigma.v.star)

      # Generate observational data
      x0 0)
      y0 0) * x0

      # Generate interventional data
      x1 <- rep(1, N)
      y1 0) * x1

      # Construct design matrix for electability calculations
      design <- cbind(z.star, u.star, v.star)

      ### Analysis ###

      # Estimate Andrew's proposal (using nonparametric regression)
      index <- which(x0 == 1)
      n0 <- xgb.DMatrix(data = design[index, ], label = y0[index])
      gbm.cv <- xgb.cv(data = n0, nfold = 10, nrounds = 500, objective = "binary:logistic", verbose = FALSE)
      lambda <- which.min(gbm.cv[["evaluation_log"]][, "test_error_mean"][["test_error_mean"]])
      m0 <- xgboost(data = n0, nrounds = lambda, objective = "binary:logistic", verbose = FALSE)
      n0 <- xgb.DMatrix(data = design, label = y0)
      p0 <- predict(m0, n0)

      # Estimate Ram's proposal (using nonparametric regression)
      n1 <- xgb.DMatrix(data = design[, c(1, 3)], label = y0)
      gbm.cv <- xgb.cv(data = n1, nfold = 10, nrounds = 500, objective = "binary:logistic", verbose = FALSE)
      lambda <- which.min(gbm.cv[["evaluation_log"]][, "test_error_mean"][["test_error_mean"]])
      m1 <- xgboost(data = n1, nrounds = lambda, objective = "binary:logistic", verbose = FALSE)
      p1 p1)

    • Attempt II, part I of II:

      # install.packages(“xgboost”)
      library(xgboost)

      ### Specification ###

      # Number of simulations
      N = 100000

      # Means of exogenous random variables
      mu.z = 0
      mu.z.star = 0
      mu.u = 0
      mu.u.star = 0
      mu.v = 0
      mu.v.star = 0

      # SDs of exogenous random variables
      sigma.z = 1
      sigma.z.star = 1
      sigma.u = 1
      sigma.u.star = 1
      sigma.v = 1
      sigma.v.star = 1

      # Coefficients for determining nomination status
      a0 = -2
      a1 = 2
      a2 = 1
      a3 = 1
      a4 = 1

      # Coefficients for determining election status
      b0 = 2
      b1 = 2
      b2 = 1
      b3 = 1
      b4 = 1

      ### Simulation ###

      # Generate exogenous random variables
      z = rnorm(N, mu.z, sigma.z)
      z.star = rnorm(N, mu.z.star, sigma.z.star)
      u = rnorm(N, mu.u, sigma.u)
      u.star = rnorm(N, mu.u.star, sigma.u.star)
      v = rnorm(N, mu.v, sigma.v)
      v.star = rnorm(N, mu.v.star, sigma.v.star)

      # Generate observational data
      x0 = 1 * (a0 + (a1 * z) + (a2 * z.star) + (a3 * u) + (a4 * u.star) > 0)
      y0 = 1 * (b0 + (b1 * z) + (b2 * z.star) + (b3 * v) + (b4 * v.star) > 0) * x0

      # Generate interventional data
      x1 = rep(1, N)
      y1 = 1 * (b0 + (b1 * z) + (b2 * z.star) + (b3 * v) + (b4 * v.star) > 0) * x1

      • Another typo… The intended coefficients for the war with Canada scenario are:

        a0 = -2
        a1 = 2
        a2 = 1
        a3 = 1
        a4 = 1

        b0 = -2
        b1 = 2
        b2 = 1
        b3 = 1
        b4 = 1

        This implies that the average candidate in this population is very unlikely to be nominated, and even if nominated very unlikely to be elected (hence low electability). However, if there is a war with Canada (also very unlikely), the candidate is both more likely to be nominated, and more likely to be elected. In this scenario, (my understanding of) Andrew’s proposal judges the average candidate to be somewhat electable (~60%), while my proposal judges them to be rather unelectable (~25%).

    • Attempt II, part II of II:

      ### Analysis ###

      # Construct design matrix for electability calculations
      design = cbind(z.star, u.star, v.star)

      # Estimate Andrew’s proposal (using nonparametric regression)
      index = which(x0 == 1)
      n0 = xgb.DMatrix(data = design[index, ], label = y0[index])
      gbm.cv = xgb.cv(data = n0, nfold = 10, nrounds = 500, objective = “binary:logistic”, verbose = FALSE)
      lambda = which.min(gbm.cv[[“evaluation_log”]][, “test_error_mean”][[“test_error_mean”]])
      m0 = xgboost(data = n0, nrounds = lambda, objective = “binary:logistic”, verbose = FALSE)
      n0 = xgb.DMatrix(data = design, label = y0)
      p0 = predict(m0, n0)

      # Estimate Ram’s proposal (using nonparametric regression)
      n1 = xgb.DMatrix(data = design[, c(1, 3)], label = y0)
      gbm.cv = xgb.cv(data = n1, nfold = 10, nrounds = 500, objective = “binary:logistic”, verbose = FALSE)
      lambda = which.min(gbm.cv[[“evaluation_log”]][, “test_error_mean”][[“test_error_mean”]])
      m1 = xgboost(data = n1, nrounds = lambda, objective = “binary:logistic”, verbose = FALSE)
      p1 = predict(m1, n1)

      ### Results ###

      # Summarize Andrew’s electability scores
      summary(p0)
      sd(p0)

      # Summarize Ram’s electability scores
      summary(p1)
      sd(p1)

      # Summarize paired differences
      summary(p0 – p1)
      sd(p0 – p1)

      # Proportion of simulations where Andrew’s score exceeds Ram’s
      mean(p0 > p1)

      • Typo…

        # Estimate Ram’s proposal (using nonparametric regression)
        n1 = xgb.DMatrix(data = design[, c(1, 3)], label = y0)

        should instead say

        # Estimate Ram’s proposal (using nonparametric regression)
        n1 = xgb.DMatrix(data = design[, c(1, 3)], label = y1)

  16. Apologies for the mess of comments above. I uploaded a PDF to Dropbox instead:

    https://www.dropbox.com/s/pudrjf94thz36fm/Electability.pdf?dl=0

    This defines notation, lays out two models as both DAGs and structural equations, provides (my understanding of) Andrew’s and my definitions of electability in terms of these models, and includes a (typo-free!) R script for illustrating differences between our definitions, with a default specification designed by analogy with Carlos’s war with Canada example.

    Would appreciate feedback from anyone (still!) interested in this discussion. The script ought to fully disambiguate my prior comments, but let me know if I can clarify anything (or if the link doesn’t work).

    • Fascinating. I have no idea how to read these graphs! Even though we have been arriving at similar conclusions, it seems we have been following very different approaches.
      Is there a guide to understanding this graph notation that you can recommend?

      The causal graph examples I gave are the standard DAG notation, for example found in Pearl’s
      “Causality” and Pearl, Glymour, and Jewell’s “Causal Inference in Statistics: A Primer.”

      • Ha, that’s funny. The difference between my notation and Pearl’s is mostly due to my erroneous recollection. What I mean by the observational model graph is this:

        (1) Nomination (“x”) is a parent of election (“y”);
        (2) Some stuff (“z”) is a parent of both nomination and election;
        (3) Some stuff (“u”) is a parent only of nomination;
        (4) Some stuff (“v”) is a parent only of election;
        (5) Some stuff (“*”) is public information, and so is fully conditioned on by efficient markets.

        Items (1)-(5) are equivalently reflected in the structural equations for x and y. To get the interventional model graph, we delete the arrows going into x, and replace the random variable x with the constant x = 1. In the equations, this means replacing the structural equation for x with the constant 1, but otherwise leaving the rest of the equations unmodified. Does that help?

        I think the only substantive difference between your conception and mine is that I’m distinguishing between public and private information, since efficient markets fully condition on the former but not the latter. Otherwise I think we’re telling the same story conceptually, which is why we arrive at the same conceptual conclusion. But I’d be grateful to be retaught how to write these DAGs and equations to conform to more standard notation!

      • To turn my observational model DAG into your Figure 1, just collapse z*, z, u*, u, v* and v into one variable. This is what you call “background”. It’s the same model, I’m just drawing some further distinctions between components of “background” which aren’t super important from a conceptual POV, but which are useful for motivating the simulation script.

        • Ah, I get it now. Thanks!! Great to see this programmed out in R.

          I hadn’t considered separating out public and private variables. That’s an interesting idea and, as compared with my examples, it engages more with the prediction market source of the p(win | nominated) values. One point it made me wonder about is whether there should be arrows between the public and private variables.

          It’s been a pleasure to have these discussions about electability, especially since I have difficulty finding good papers on it in the social choice and political science literatures. On the one hand, many peoples’ main political objective is to beat the other party’s presidential candidate in the general election. Yet, on the other hand, there seems to be relatively little theoretical or evidence-based research on how you should cast your primary vote, strategically speaking, if your main goal is to beat the opposing party’s candidate. Or maybe I’ve just missed the articles in my searches.

        • Thanks. Given all of the misunderstandings in the earlier thread, I figured a program was less subject to misinterpretation.

          Regarding public v. non-public information, I do think it’s important because it complicates the simplest version of the correlation != causation story, namely one in which the only confounder is public information. In that case, the response is that that market is conditioning on it, so should handle this confounder just fine. The real problem is unobserved confounding, which means nonpublic information. This could be either time-invariant factors that have not yet been disclosed, or they could be future values of time-varying factors (which are thus presently unknown).

          I did not draw arrows between the public and nonpublic variables because I’m not interested in modeling interrelations among them. They exist in the graph as a partitioning exercise. E.g., the nonpublic information that affects both nominations and elections does not include consequences of public information that affects both nominations and elections. Rather, it only includes further exogenous information not reflected in the public information variable. So while there are unobserved variables on the path from z* to x and z* to y, conceptually these are not included in z. This allows me to draw the graph the way I did.

          Agree completely on the literature. I’m sort of surprised we’re having this discussion at such a primitive level—it ought to be easy to cite papers on this, but I’m not aware of any. Would appreciate pointers to relevant references if anyone finds any.

Leave a Reply

Your email address will not be published. Required fields are marked *