Thomas Hühn writes:
I’m thinking about doing a Bayesian analysis of a very small subset of PISA or TIMSS data. Those large-scale education surveys do not report students achievement scores as single numbers, but they report five or ten numbers, so called plausible values. Those plausible values have been sampled from a constructed probability distribution.
The user guides and methodology papers strongly warn against taking those five plausible values as five observations, and also against taking the mean of those five plausible values and doing statistical analysis on that.
Instead you’re supposed to do five analyses on each set of plausible values and then take the arithmetic mean of the descriptive statistics metrics that you care about. Also, you’re supposed to calculate some imputation error, but I haven’t really understood that, yet.
How would a Bayesian proceed here?
I guess they would also do five analyses and then end up with five posterior probability distributions. How can they be combined? It surely cannot be as easy as doing a vector add in R and dividing by five (you’re sensing my small glimmer of hope here)?
My response: I’m not sure. I guess it depends where the plausible values come from? The method they describes sounds similar to the analysis of multiple imputations. There can be reasons that such an analysis will work better than taking the average for each person, because if you take the average for each person, your analysis kinda thinks that people are less variable than they really are.
One way to get a handle on this would be to simulate fake data based on a known model.
https://www.oecd.org/pisa/data/httpoecdorgpisadatabase-instructions.htm
I couldn’t figure out what exactly they are doing, but there is a link for anyone willing to put in the effort. The description goes on for a bit longer.
It seems *way* too byzantine for messy data like this imo. Why not just report the test scores without all the assumptions?
I agree that it seems “way too byzantine” but I reach the opposite conclusion. If they just reported the test scores, they would be asking you to trust all those myriad assumptions and adjustments they actually did to get those scores. I actually want more detail so that I can understand whether their test scores are meaningful – the procedure appears to result from students answering random subsets of test questions. Since this introduces potential measurement bias, I’m reluctant to simply trust that the scores that they are reporting are reliable. Of course, they are all experts with the right degrees and showing pages and pages of documentation, but I thought we were past the point of trusting them on that basis.
(like you, I am not interested enough to wade through all that, but I believe it needs to be there)
Then the raw data seems to be what questions each student was asked, in addition to their answers. It seems like that is available:
I’ll try to estimate an upper bound on the amount of data.
Assumptions:
– There are 7 billion people
– 1% are the age to take the test (15 years old I gathered)
– Each student answers 100 questions
– Each answer has 5 metrics associated with it
That gives 7e9*0.01*100*5 = 35e9 datapoints.
Further assume:
-Each datapoint is a 64 bit (8 byte) value
So 35e9*8 = 280 GB of data. That is quite a bit, but multi-TB servers exist. So it should be able to all fit in memory at the same time. Really it probably doesn’t need to be in RAM at once, at least some of the data could be stored as 32/16 or even 2-bit, and that is supposed to be an upper bound.
I don’t think its unreasonable for researchers to do their own analysis on this data. Its not clear to me what question they want answered, but that is a different issue.
this is all quite comical! It strikes me as something akin to creating a fake data set and mining the noise within it – or something like that? Who knows really WTH this is achieving if anything, other than keeping data analysts busy.
It would be interesting if it was shown somewhere that any of this has any meaning beyond what could be achieved by asking all the students all the same questions and using relatively simple statistical analysis (e.g., mean mode median sigma), and if such a demonstration could be achieved with language and graphics that a reasonably STEM-educated person could comprehend – e.g., without recourse to thick layers of statistical mumbo jumbo and graphics with multiply indexed indexes of indices plotting ratios of distributions of distributions against other distributions of ratios of distributions, so ridiculously confounded that even a statistician has no idea what’s going on (which frequently seems to be the case).
It seems like at the core they collected interesting data about test-taking behavior.
But yea, I’d definitely start with the raw data first. Then maybe there would be a bunch of times I checked their docs and was like “oh, so thats why they did it that way.”
I have no doubt garden of forking paths would show up here and you can get totally different results from the same dataset.
Perhaps they used their model to predict future data accurately, and that is why we should trust it? But in that case I’d guess they would be (rightfully) bragging about it on the summary page.
It would be hard to do with many off-the-shelf analysis packages, but if you’re comfortable programming your own likelihoods in some PML it seems completely sensible to me to take the expected likelihood over the samples they provide ? So if for observed values y_1 … y_n you would normally use a likelihood p_1(y_1 | theta) * … * p_n(y_n | theta) with parameters/latent variables theta, it would be logical to use instead ( 1/5 (p_1(y_11 | theta) + … + p_1(y_15 | theta)) * … * ( 1/5 (p_n(y_n1 | theta) + … + p(y_n5 | theta)) if you have five values y_i1…y_i5 for each observation i. Then maximize/sample your likelihood/posterior as usual. Curious to see if anyone can think of something that would go wrong. I have no domain knowledge on these studies.
I don’t this is as exotic as it’s being made out to be. The calculation of conditional plausible values in large-scale assessments was pioneered by psychometrician Robert Mislevy based on Little & Rubin’s work on the use of multiple imputations to estimate missing data. Individual test-takers never take all of the items in these assessments, as the desired domain coverage exceeds what’s practical for a test-taker—typically at most a couple of hours worth of items. Test items typically are arrayed in balanced incomplete blocks. The assessments are not designed to produce reliable individual scores, but rather to generate unbiased population edtimates.
Agreed.
Thanks for you comment. To add some confusion to complexity of this topic. Not only the test booklets are randomly assign to individual students but even the whole test domain are not completed by everyone, for example about 2/3 of all students (within each country) complete some test booklets of secondary domain (in PISA 2022 it was reading or science) the rest 1/3 had no test items, yet everyone has imputed plausible values for every domain.
I guess the issue is whether you can explain what they did in a few paragraphs. Like essentially point out all the major assumptions being made.
So it says their estimates are from an IRT (Item Response Theory) model. So instead of an analysis where you infer how capable each student is be just summing up how many correct they got, and making sure each student gets a form of equivalent difficulty, the estimates of student ability are estimated item by item. If you know the difficulty of each item, then whether or not they got answered that question correctly gives you information on the ability of the student. The tricky thing is there are two inter-dependent things to estimate. Estimates of how difficult items are are dependent on how well students of varying capabilities do each item. Estimates of how capable each student is depends on how well they do on items of varying difficulties. So, I think the multiple imputation in this case is an attempt to take into account varying estimates of the difficulties of the test items? And I think that they only do a few estimates because it’s pretty computationally intensive to estimate the IRT model in the first place. But it does seem awkward. It would be more useful in some ways to estimate the IRT model plus whatever else you want to analyze at the same time. But I’m guessing the thought is that this is good enough for many analyses, and better than not acknowledging at all that there is uncertainty about the item difficulties?
In IRT, Item difficulties (scale locations) are estimated independently from student abilities – the estimates of ability and difficulty are not inter-dependent.
Another point that is relevant to your comment, is that parameter estimates are based not only on the item responses (i.e. the answers to test questions); they are conditioned on group variables and latent regressors (brief explanation and link to PISA technical report, here: https://www.edmeasurementsurveys.com/IRT/conditioning-variables-for-marginal-maximum-likelihood-estimation.html#conditioning-variables)
The short answer is that PVs are necessary to avoid biased estimates of individual abilities.
Here’s a good introductory-level primer:
https://www.edmeasurementsurveys.com/IRT/Ch14.html#why-do-we-need-plausible-values
Hang on Andrew…
It sounds as though the datasets provided have been multiply imputed, in which case the warnings are sensible. But hey, if you don’t want go through with Rubin’s rules, you can do Bayesian analysis of multiply imputed data.
There’s a 2010 paper by Zhou and Reiter on Bayesian inference using multiply-imputed data, and they attribute the idea to a 2004 book by Gelman, Carlin, Stern & Rubin (good to know I’m not the only one who forgets things I’ve written)! Anyway, the idea is cool: go ahead and do your Bayesian analysis within each imputed dataset then just chuck all the posterior samples together at the end. None of the faff of having to separate within- and between-imputation variation in Rubin’s rules.
X Zhou, JP Reiter (2010). A note on Bayesian inference after multiple imputation. The American Statistician. 64(2):159–163 https://www2.stat.duke.edu/~jerry/Papers/tas10.pdf
A Gelman, JB Carlin, HS Stern, DB Rubin (2004). Bayesian Data Analysis, CRC press.