Ralph Blair sent this in. It’s so horrible that I have to put it in the continuation part of the blog entry. I recommend you all stop reading right here.
Stop . . . It’s not too late!!!!!!!!!!!
OK, here it is. No, no, no… (Here’s the technical article explaining the statistical flaws in this stuff.) Mistakes are made all the time, of course, but it doesn’t help when they are tied to wacky political agendas.
The news article begins:
The Beautiful Person club is an exclusive one, and entry brings much – fame, wealth … and daughters. Think of the most beautiful couples in the world – they all have daughters. Tom Cruise and Katie Holmes? Check. Denise Richards and Charlie Sheen? Check. Brangelina and Bennifer? Check and check.
Actually, we looked up a few years of People Magazine’s 50 most beautiful people, and they were as likely as anyone else to have boys:
One way to calibrate our thinking about Kanazawa’s results is to collect more data. Every year, People magazine publishes a list of the fifty most beautiful people, and, because they are celebrities, it is not difficult to track down the sexes of their children, which we did for the years 1995–2000.
As of 2007, the 50 most beautiful people of 1995 had 32 girls and 24 boys, or 57.1% girls, which is 8.6 percentage points higher than the population frequency of 48.5%. This sounds like good news for the hypothesis. But the standard error is 0.5/sqrt(56) = 6.7%, so the discrepancy is not statistically significant. Let’s get more data.
The 50 most beautiful people of 1996 had 45 girls and 35 boys: 56.2% girls, or 7.8% more than in the general population. Good news! Combining with 1995 yields 56.6% girls—8.1% more than expected—with a standard error of 4.3%, tantalizingly close to statistical significance. Let’s continue to get some confirming evidence.
The 50 most beautiful people of 1997 had 24 girls and 35 boys—no, this goes in the wrong direction, let’s keep going . . . For 1998, we have 21 girls and 25 boys, for 1999 we have 23 girls and 30 boys, and the class of 2000 has had 29 girls and 25 boys.
Putting all the years together and removing the duplicates, such as Brad Pitt, People’s most beautiful people from 1995 to 2000 have had 157 girls out of 329 children, or 47.7% girls (with standard error 2.8%), a statistically insignificant 0.8% percentage points lower than the population frequency. So nothing much seems to be going on here. But if statistically insignificant effects with a standard error of 4.3% were considered acceptable, we could publish a paper every two years with the data from the latest “most beautiful people.”
I don’t blame the reporter (Maxine Shen) for this: it’s natural to believe something that’s been published in a book and a scientific journal. Perhaps, though, someone could send a note to whoever reviews this sort of book so that the errors won’t be propagated indefinitely??
You seem to be saying that mistaken ideas — that can be shown to be wrong — do more harm than good. Whereas I think that correctness is only one of several important dimensions. Another is how interesting and/or practical and/or down-to-earth the idea is. The more interesting etc. an idea is, the more motivating it is. I think we need motivation — people wanting to do science — even more than we need correct ideas.
You write, "You seem to be saying that mistaken ideas — that can be shown to be wrong — do more harm than good."
No, I'm making no such general claim. I'm talking about the specific claims made in the New York Post article, which are mistakenly presented as supported by scientific research.
People can make whatever claims they want–go to any schoolyard fully of 7-year-olds and you can get lots of theories about sex differences. I don't think it is a good idea to say that these claims are supported by data when they're not.
To put it another way: if speculation about sex differences were rare, then I agree that Kanazawa could be making important contributions with his claims. Actually, though, such speculations are commonplace, and Kanazawa's key contribution or "value added" is in providing statistical evidence. His evidence is his ticket to getting his claims taken more seriously than anybody else's speculations, and if, as it happens, the evidence is not there, I think people should know. Science is about data, not just making statements that sound good.
Andrew and Seth,
Once it is printed in the New York, it is officially a rumor. Many newspapers and TV news shows around the world will pick this up and make it a common wisdom. Let us hope it does not result in some type of eugenics program of dubious kind somewhere. At this point, the only way to expose the data as flawed and possibly false is to put your text on snope.
There are plenty of people going into science, and in my experience, a significant percentage passing through and out the other side because there are not enough positions at the other end. Do we really need more?
Basically I'm asking a question: Where does this work fall on other dimensions that matter?
Andrew says this work is "so horrible" but how he has arrived at this view is not clear. Okay, it's wrong; but what about dimensions other than correctness?
1. I didn't say the work is so horrible; I said the news report is so horrible.
2. As I noted in comment above, these sorts of speculations about sex differences are commonplace. What Kanazawa is adding id data analysis. Since the data don't particularly support his claims, I don't see that he's adding anything.
To summarize: 2 dimensions, concept and data analysis. Concept is not new, data analysis is not correct.
What bothers me is not this particular work–we all know that statistics is hard, and many people make statistical mistakes, some of which inevitably make it into scientific journals–but that it is misleadingly reported as fact to the general public.
I'd say that
2. data analysis (including the data)
are two parts of the thing (scientific paper) being judged. So far you've mentioned two dimensions of evaluation:
It is becoming clearer to me why you judge the paper as you do. I think at least four other dimensions of evaluation are important:
3. practical value
4. ability to inspire future research
5. ability to make people think
6. ability to attract attention
A good methods paper is high on #4, for example. #6 is about what Brian Wansink calls "cool data" — the "cool data" factor.
I can't tell from your comments on this paper whether you use these other dimensions to judge papers. Since science is obviously about more than being right or wrong (or doing an incorrect or correct data analysis), I'm sure you use other dimensions; I just can't tell what they are.
I just don't see anything special about Kanazawa's work, _except_ for its instructive statistical errors. Of course someone can play contrarian and say that the work has value anyway, but I see no evidence for that. Maybe Rosie Ruiz would've won the Boston Marathon for real if she'd run the whole way, but we'll never know. Maybe that Canadian dude, Dr. Chandra, found real things, despite the errors in his papers. In all seriousness, if you're interested in the topic of sex ratios, I suggest you read some of the more serious papers published in the area. There's no particular reason to start with a series of papers whose claim to fame is that they got press by claiming to find things that weren't in the data.
Kanazawa's work is certainly on a more "cool" topic than most scientific work. Perhaps it is more thought-provoking. Apparently you found it so. My interest is not in sex ratios; it is how scientific work is judged. Mostly I hear too much about "right" and "wrong" and too little about other qualities. This puzzles and interests me; I wonder why it is.
Chandra's work is in a whole different universe of incorrect than Kanazawa's work.
The "coolness" of Kanazawa's work comes because of the presumption that it is correct. Cool-sounding statements are easy to come up with; what's hard is to make a cool statement that is actually supported by evidence.
For example, here's a cool and noteworthy story:
I was walking down the street and I saw 3 guys unicycling, each with two monkeys on their shoulder.
Well, it would be cool if it really happened. Actually, I just made it up.
That's how I feel about Kanazawa's papers. He didn't make things up but he made mistakes, leading him to interpret patterns in data that could easily be explainable by chance. Recall that Kanazawa's claims got publicity not just because they were "cool" but because they were apparently supported by evidence. Without the evidence, you could get equally cool findings any day of the week by running correlations on publicly available surveys.
OK, here's another example:
People with first names beginning with A through M have IQ's that are 10 points higher, than average, than people with first names beginning with N through Z. Wow–that's cool! Maybe it has something to do with where these kids sit in the classroom, or maybe it has something to do with who gives kids names in different parts of the alphabet, or maybe something with ethnic groups . . . .
No, not really. I just made it up. But, don't worry, I won't get publicity for this one–for one thing, it wasn't published in a peer-reviewed journal.
There is a large body of research relating to sex ratios at birth. Among the possible contributors to altering the sex ratio:
Time of insemination, Coital rate (see JF Martin, 1997),
Polygyny (see JWM Whiting, 1993),
Residential arrangements (see K Norberg, 2004),
and Female Circumcision (in submission).
While I assume we can omit polygyny and female circumcision from the discussion of the 50 most-beautiful people, one must control for these other factors if you are going to look for beauty as a contributor.
My impression from reading the literature and talking to people is that sex ratio variation is small (less than 1%) except in pretty extreme situations such as famine. The point of my paper with David Weakliem is that you can pretty much forget about discovering effects of this size from the sorts of samples that Kanazawa was using.