David Weakliem is a sociologist (and coauthor of the article Of beauty, sex, and power: statistical challenges in estimating small effects). For the past decade he’s maintained a blog, Just the social facts, ma’am, that’s full of . . . well, facts. Mostly summaries of public opinion polls. Summarizing poll results might seem pretty easy, but it’s not.
Here are his three most recent posts at the time of this writing:
I read the one on institutional trust, and I don’t think it’s safe to just take the numbers at face value. The data was gathered by Gallup, which I’m sure in the eyes of many, falls into some category of “institution”. I suspect there’s some pretty severe non-response bias going on there as a result.
I don’t begrudge Weakliem (or anyone else) reporting and discussing data like this, but I’d like more visible cognizance of potential major flaws.
The problem with any set of numbers on the issue of abortion is that a large cohort on one side is more interested in sexual promiscuity than babies. These are folks who feel that an unwanted baby is a just price that women should be forced to pay for promiscuity. (If you are bewildered by these same anti-abortion folks being anti-birth control…well, I just explained why that is so.)
I would be very interested in a clever survey that teases these two attitudes – genuine compassion for someone else’s fetus, vs. envy of the sex lives of the promiscuous – apart.
The problem with any set of numbers on the issue of abortion is that a large cohort on one side is more interested in sexual promiscuity than babies. These are folks who feel an unwanted baby is too high a price for women to be forced to pay for promiscuity. (If you are bewildered by these same pro-choice folks being pro-birth control…well, I just explained why that is so.)
I’m not making the above argument, just pointing out the remarkable symmetry!
From my perspective, summaries of public opinion polls are not actually “facts”.
I mean, yeah if 38.4 percent of respondents to a certain survey chose “Strongly Agree” on Item 27 then that 38.4 percent number is a factual statement about that survey. I call that a “factoid”.
But survey researchers need to be ever vigilant against the temptation to reify survey responses into factual information regarding objective reality.
This is still true of every measurement. If your volt meter reads 13.3V it’s a true fact that that’s what your volt meter reads… but it doesn’t mean the voltage on the two terminals is 13.3000000000 V in reality. The difference is that at least Voltage is a well defined concept. When it comes to public opinion what does “strongly agree” even mean, or how about what do the words in the question they’re “strongly agreeing” mean, probably something slightly different for different people.
So the conceptual issue is bigger with public opinion polls, but even with say physical measurements this conceptual issue is there. Some fluids don’t have a well defined viscosity (they have a nonlinear relationship between shear rate and shear stress). Some materials don’t have a fixed “notch toughness”. Concrete samples even from the same concrete mixer don’t have a fixed compressive strength… etc etc. It’s worthwhile to remember that even outside of the more fuzzy domains.