The Groovy Grandma Problem in survey research

This came up in our Xbox survey and I’ve used the phrase many times but not in writing. I’m posting it here so we have something to point to when we include it in the lexicon.

Here’s the story. We asked some political survey questions to a few hundred thousand users of the Microsoft Xbox videogaming platform. Most survey samples overrepresent women and old people. The Xbox survey was different it overrepresented young men. It was a huge sample, though, so it still contained enough old women to cover that demographic segment of the population.

The concern was not sample size but rather representativeness within the demographic slice. Yes, our survey included many old women (even if not in proportion to their share of the general population of voters), but they were not typical old women. They were videogame players–or, at least, there they were on the Xbox console. We called this the “groovy grandma problem”: we’re poststratifying on age and sex but not on grooviness, or on the propensity to spend time playing videogames. (Also maybe the respondents to the survey questions weren’t the same people who’d given the demographic information, but that’s another issue.)

In the case of the Xbox survey, I don’t think this was a big problem–even if these participants were different on average from women of that age category in the population, we saw no reason to believe that they were much different in their political attitudes. Had we been asking questions about their leisure activities, it would’ve been a different story.

In any case, “the groovy grandma problem” is a convenient way to point to this concern about conditional nonrepresentativeness.

16 thoughts on “The Groovy Grandma Problem in survey research

  1. Instead of

    “Yes, our survey included many old women”

    perhaps it would sound less harsh to write

    “Yes, our survey included many elderly women”

    Likewise,

    “but they were not typical old women”

    “but they were not typical elderly women”

    On the other hand, I suppose some might find the alternatives too subtle to be meaningful.

  2. I find this a more serious problem than what you state. Sure, if the questions are about leisure activities, the non-representativeness of the elderly women in the sample is a problem. But if the question is about political attitudes, then it is still a strong assumption to believe those attitudes are independent of participation in videogames. I would think the default assumption should be that they are related – and then to make explicit assumptions (probably several different ones) about the nature of the dependence and see how that impacts the results. Assuming independence is a convenient way to avoid the issue. This seems uncharacteristic for Andrew, who usually reminds us to be explicit about our modeling assumptions. Elderly women playing videogames seems to me a quite unusual demographic and one that might differ from all elderly women in almost all ways. At least that would be my default assumption.

    • I agree.

      Another potentially serious problem:

      Among xbox survey responders who say they are 18~29 year old males, I’m guessing ~99% of them are actually 18~29 year old males.
      Among xbox survey responders who say they are 65+ year old females, I’m guessing the proportion of actually 65+ year old females is going to be much lower.

      Somewhat related to Lizardman’s constant. (trolling and/or mistake)

      • Something like that was my thought: namely that the number of elderly females who are actually Xbox users has to be real small.

        But. How about grandparents who have grandkids that play with an Xbox and (being a doting grandparent) granny tries it out. Is our hypothetical granny here an “Xbox user”, and might there be more of such folks than I imagine???

        Dunno. It still seems like its really gotta be a really tiny set.

  3. I would think that mischievous responding could be a pretty big concern in that survey. So not necessarily groovy grammas, but just kids who find it amusing to answer the demo questions by saying that they’re 72 YO women. Did you have any way to assess whether the demographic reports are trustworthy?

Leave a Reply

Your email address will not be published. Required fields are marked *