Huib Meulenbelt writes:
Assume we have the following hierarchical Bradley-Terry model:
data { int<lower=0> K; // players int<lower=0> N; // number of rallies int<lower=1, upper=K> player_a; // player a int<lower=1, upper=K> player_b; // player b int<lower=0> y; // number of rallies won } parameters { real<lower=0> sigma; vector[K] skill; // ability for player K } model{ sigma ~ lognormal(0, 0.5); skill ~ normal(0, sigma); y ~ binomial_logit(N, skill[player_a] - skill[player_b]); }In this blog post you argue “there’s a lot of information in the score (or vote) differential that’s thrown away if you just look at win/loss.”
I agree completely. Of each rally I obtained the length of the rally and I would like this variable to influence the skill level of the player and opponent. The skill level of the two players are closer to each other when the game ends in 11-7 and the match lasts 500 seconds than when the game ends in 11-7 and the match only lasts 100 seconds.
So, we move from
p[player A wins over player B] = logistic(skill_A – skill_B)to
p[player A wins over player B] = logistic(f(rally_length) * (skill_A – skill_B))
How would you define this function?
My reply: This is known in psychometrics as an item-response model with a discrimination parameter. The multiplier, f(rally_length) in the above notation, is called the discrimination: the idea is that the higher it is, the more predictive the skill-level difference is of the outcome. If discrimination is zero, the skill difference doesn’t predict at all, and a negative discrimination corresponds to a prediction that goes in the unexpected direction (the worse player being more likely to win).
My answer to the immediate question above is: sure, try it out. You could start with some simple form for the function f and see how it works. Ultimately I’m not thrilled with this model because it is not generative. I expect you can do better by modeling the length of the rally as an intermediate outcome. You can do this in Stan too. I’d recommend starting with just a single parameter per player, but you might need to add another parameter for each player if the rally length varies systematically by player after adjusting for ability.
But the biggest thing is . . . above you say that you agree with me to model score differential and not just win/loss, but then in your model you’re only including win/loss as an outcome. You’re throwing away information! Don’t do that. Whatever model you use, I strongly recommend you use score differential, not win/loss, as your outcome.
It looks to me like each “item” (sorry, psychometrics brain) is a single rally instead of a complete match. In the proposed model with discriminations, the function of rally length will weight each observation by placing more importance on rallies decided quickly. This might not do the thing you want to the eventual skill estimates.
Also the model is making a strong unidimensional assumption about player skill that might be at odds with the form of discrimination being introduced into the response process. There are other ways in the psychometric literature to conceptualize latent ability and speed that may be a better fit for including rally length in a measurement model (a good place to start is this paper from Wim Van Der Linden, https://link.springer.com/article/10.1007/s11336-006-1478-z).
What is the definition of “generative model”? For this model, it seems possible to generate parameters from the prior and then data from the likelihood.
Ed:
I might be misunderstood, but I had the impression that Meulenbelt wanted to condition on rally length. A generative model would generate rally length.
Also, just a minor point: You wrote of generating data from the likelihood. That’s not correct. You can generate data from the data model, p(y|theta). The likelihood is defined as a function of theta that is proportional to p(y|theta). Knowing the likelihood does not tell you the data model.
The question of what’s a generative model reduces to the question of what is data. In the terminology of Andrew and Jennifer’s regression book, the question is whether an observed quantity is modeled or not. We typically don’t model the size of a data set, for example, but nobody then says, “Hey, your model isn’t generative.” Here, Andrew’s assuming that the data is rally length, opponents, and winner and rally length isn’t being modeled.
In a regression of outcome y on covariates x, we typically don’t model x. But if we did, the marginal inference on regression parameters wouldn’t change (there’s a section in Bayesian Data Analysis deriving this). So we usually don’t bother modeling predictors unless we have missing data. Nevertheless, the ML folks call a regression a “discriminative model” because it doesn’t (typically) include a model for x.
Bob,
Yeah, it seems strange to me to not model rally length in this example, as it’s an intermediate outcome being used in the model. I can easily imagine a model that doesn’t account for rally length at all, but once it’s there as a predictor, it seems only natural to consider it modeled data.
The Stan program here is using a centered parameterization for the hierarchical model. I would recommend trying a non-centered parameterization, which you can achieve by declaring skill as follows.
What Andrew’s suggesting is the Bradley-Terry equivalent of the IRT 2PL educational testing model, where there’s a difficulty and discimrination parameter per test question. When the discrimination is a free parameter, it can be difficult to fit, but if your discrimation is a function of data, then it should be easier (just be sure to make sure however it fits, you constrain the discrimination to be positive). The 3PL model adds a chance of guessing the right answer on a test by chance (e.g., multiple choice).