In a unit about the law of large numbers, sample size, and margins of error, I used the notorious beauty, sex, and power example:
A researcher, working with a sample of size 3000, found that the children of beautiful parents were more likely to be girls, compared to the children of less-attractive parents.
Can such a claim really be supported by the data at hand?
One way to get a sense of this is to consider possible effect sizes. It’s hard to envision a large effect; based on everything I’ve seen about sex ratios, I’d say .005 (i.e., one-half of one percentage point) is an upper bound on any possible difference in Pr(girl) comparing attractive and unattractive parents.
How big a sample size do you need to measure a proportion with that sort of accuracy? Since we’re doing a comparison, you’d need to measure the proportion of girls within each group (attractive or unattractive parents) to within about a quarter of a percentage point, or .0025. The standard deviation of a proportion is .5/sqrt(n), so we need to have roughly .5/sqrt(n)=.0025, or n=(.5/.0025)^2=40,000 in each group.
So, to have any chance of discovering this hypothetical difference in sex ratios, we’d need at least 40,000 attractive parents and 40,000 unattractive—a sample of 80,000 at an absolute minimum.
What the researcher actually had was a sample of 3000. Hopeless.
You might as well try to weld steel with a cigarette lighter.
Or do embroidery with a knitting needle.
OK, they’re not the best analogies ever. But I avoided sports!
This was actually illuminating for me. If someone goes through life with the same level of understanding of sports as my level of understanding of embroidery they would be at a real disadvantage. (Sure, I could figure out from context that knitting needles are too big and clunky for embroidery, but that’s mainly because I already understood the statistical message and used that as an analogy to understand embroidery.) Analogies really help solidify messages and if the embroidery one were the only analogy there this message would have gone unsolidified for me as I’m sure many messages go unsolidified for people who don’t follow sports. I knew this intellectually already, but this example made me feel it. Too bad it’s often so easy to find a great sports analogy.
Actually, what Z said is interesting. Because the reason you can’t do embroidery with a knitting needle isn’t really that a knitting needle is big and clunky. It’s that it doesn’t have an eye to hold the thread – you can’t do any sort of sewing with it at all. So was that the intended message from Andrew, or did he come up with the same sort of analogy in the sewing world that I would come up with in the sporting world?
I have found that good analogies from a range of areas is the best way to engage people. I’ve sat through too many lectures where everyone turns off because the lecturer is a soil scientist and gives every single example of the course models from a soil science perspective, or biologists that give every example in terms of proteins and sequences.
Being such a generalist subject, I think a liberal arts background is one of the greatest allies in learning and understanding statistics. I know my opinion will likely be unpopular here, but I think far too much emphasis is placed on mathematical experience , which is rarely the problem for new players. The real challenge is understanding when a model is appropriate, how data collection methods affect outcomes, and typical inferences people draw from results and visualisations – all of which require a reasonable amount of experience in areas outside of mathematics.
Why not try using a smaller sample of ultra-attractive parents, such as movie star couples? If this effect is real, it should show up extra-strong in the world most attractive couples. For example, start with Brad Pitt and Angelina Jolie’s biological children. A few days of puttering around on IMDB and Wikipedia could find you, say, 1000 movie star couples and the names/sexes of their children.
Steve:
Amusingly enough, Weakliem and I did do this in our paper (using People Magazine’s 50 most beautiful people lists), but pretty much just as a joke. Any plausible effect size would be so small that a sample of 1000 would be essentially useless. You’d really need tens or hundreds of thousands of births to get enough data to test the hypothesis. And, at that point, the results would be very sensitive to the attractiveness measurement anyway.
The trouble with all these sex ratio studies is that the sex of babies is pretty much random. Variation in the sex ratio between groups is tiny, and much of it is explained by the simple fact that girls are hardier than boys, so when conditions get tough, the proportion of girls goes (very slightly) up, and when conditions get mild, the proportion of boys goes (very slightly) up.
To put it another way: Your idea of focusing on a sub-population with a large effect is, in general, a good one. It just doesn’t work here because the orders of magnitude are wrong. The effect size is just too small and the variability is too high.
So, why not use that as a challenge to researchers asserting this beautiful people theory: “It’s easy to find out from Wikipedia and IMDB.com the names of the children of actresses, pop singers, and the like. You could build a sample size in the thousands just from those two sources. If you don’t find a large effect among the professionally good-looking, why should we believe your theory based on your one little sample?”
Steve:
The trouble is, it often is possible to find statistical significance. Especially in a situation here with such a small effect, it would take just a small amount of selection bias to overwhelm any signal.
Rather than sending people down this rabbit hole, I’d rather recommend that researchers study more interesting, measurable phenomena.
Interestingly, there was just some recent news article this week about how in hard times more boy fetuses spontaneously abort, but the ones that do survive are apparently much hardier than average. People were trying to study this to figure out the biochemical mechanism