Lessons about statistics and research methods from that racial attitudes example

Yesterday we shared some discussions of recent survey results on racial attitudes.

For students and teachers of statistics or research methods, I think the key takeaway should be that you don’t want to pull out just one number from a survey; you want to get the big picture by looking at multiple questions, multiple years, and multiple data sources. You want to use the secret weapon.

Where do formal statistical theory and methods come in here? Not where you might think. No p-values or Bayesian inferences in the above-linked discussion, not even any confidence intervals or standard errors.

But that doesn’t mean that formal statistics are irrelevant, not at all.

Formal statistics gets used in the design and analysis of these surveys. We use probability and statistics to understand and design sampling strategies (cluster sampling, in the case of the General Social Survey) and to adjust for differences between sample and population (poststratification and survey weights, or, if these adjustments are deemed not necessary, statistical methods are used to make that call too).

Formal statistics underlies this sort of empirical work in social science—you just don’t see it because it was already done before you got to the data.

6 thoughts on “Lessons about statistics and research methods from that racial attitudes example

  1. A view from the GSS trenches:

    We knew well about the shift in 2016, but we didn’t make a big deal out of it. Perhaps a leading indicator of change, perhaps a blip. When I was writing my various papers on the 2016 election and race, I simply stayed with the more mild claim of “no evidence of increase” for items on prejudice, etc. That is enough to deal with the claim that Trump had provoked widespread white nativism. But … the moment the 2018 GSS results became available, all those variables were the first ones I checked, and I tweeted them right away:

    https://twitter.com/SLMsociology/status/1108041162420568065

    A few more tweets on multiple indicators

    https://twitter.com/SLMsociology/status/1108043493186916353

    A then it was worth showing what others were showing that the GSS hadn’t somehow lost its Republicans:

    https://twitter.com/aedwardslevy/status/1108057049458532354

    And then I did some serious work to make sure the sample was fine, and all indications are that it is.

    And then back to some tweets in response to people claiming that this is just a Democrat thing:

    https://twitter.com/SLMsociology/status/1108440368599707648

    Bottom line: We have more work to do, but this seems to fit our GSS definition of change. Something that appears once could be a blip. If twice, we start to write about it and get serious about analyzing it. Still lots to clarify, and I suspect lots of people are starting papers, but my sense is this one will stand up to further scrutiny.

      • I see my primary role as serving as a good steward of the GSS, and also writing academic articles (favoring, for the last few years open access outlets with expedited review, given my recent interest in timely topics).

        Sometimes tweeting is fun, and it was a good way to help spread the word that the GSS data were out.

        I will leave the blogging to others, like you, who have a following and who do very well!

  2. I think this is so right, which is why I right away wanted to look at the different wordings to see if they were consistent. Then I wondered if it was driven by young people. And then I looked at the weight options. It seems to be true across all regions although the middle Atlantic and Pacific were particularly huge in 2018 (jumping to above 70%).

    Then there is a larger question, that I haven’t looked at yet, but maybe Steve has, about if this part of a change about race, literally about spending in that whole series of NAT* questions (the ones I checked are up–even natspac), or something else such as a more general shift on a range of social issues.

Leave a Reply to Steve Morgan Cancel reply

Your email address will not be published. Required fields are marked *