Kaiser goes through the first chapter of Freakonomics 2 with a statistician’s reading, picking out potentially interesting claims and tracking where they come from. It’s actually the kind of review I might write–although in this particular instance I chose not to actually read the book, instead speculating on its authors’ motivations (see here, here, and here).
Here’s Kaiser:
p.20 — was surprised to learn that women used to have shorter life expectancy than men. I have always thought women live longer. This factoid is used to show that throughout history, “women have had it rougher than men” but “women have finally overtaken men in life expectancy”. I’m immediately intrigued by when this overtaking occurred. L&D do not give a date so I googled “female longevity”: first hit said “it appears that women have out survived men at least since the 1500s, when the first reliable mortality data were kept.”; the most recent hit cited CDC data which showed that U.S. females outlived males since 1900, the first year of reporting. In the Notes, L&D cite an 1980 article in the journal Speculum, published by the Medieval Academy. In any case, the cross-over probably occurred prior to any systematic collection of data so I find this minor section less than convincing.
. . .
p.29 — They cite statistics about “the typical prostitute in Chicago.” In what ways are the subjects of the study “typical” and in what ways are they not typical? The sample size was 160. They don’t say much about the selection process of the subjects, except that they all came from three South Side neighborhoods. Would like to know more about the selection.
p.30 — After much buildup, we get to their surprise: “Why has the prostitute’s wage fallen so far?” I’m looking for the data, what does it mean by “so far”? All we have is the assertion “the women’s wage premium pales in comparison to the one enjoyed by even the low-rent prostitutes from a hundred years ago.” On the previous page, we learn that modern “street prostitutes” earn $350 per week. On p.24, we learn that in the past, Chicago prostitutes took in $25 a week, “the modern equivalent of more than $25,000 a year”. Unfortunately, neither of these two numbers is comparable to $350. Dividing $25,000 by 50 weeks (approx.) gives $500 per week. So the drop is $150 off $500, or 30%. But… this is a comparison of wages from prostitution, not of “wage premium”. On p.29, the modern study found “prostitution paid about four times more than [non-prostitution] jobs.” On p.23, they say “a tempted girl who receives only $6 per week working with her hands sells her body for $25 per week” so we can compute the historical ratio as $25/$6 = 4.17 times. So, I must have gotten the wrong data.
. . .
p.46 — Some of the language is overdone. They say the men “blew away” the women in a version of an SAT-style math test with twenty questions. What does “blowing away” mean? Scoring 2 more correct questions out of 20.
. . .
The rest of the chapter — They discuss Allie, a high-end prostitute. This section has little interest for a statistician since it is a sample of one.
This last bit reminded me of my dictum that the activity we call “statistics” exists in the middle of the Venn diagram formed by measurement, comparison, and variability. No two of the three is enough.
Not a debunking
Kaiser’s comments do not represent a trashing, or debunking, of the much-criticized new Freakonomics book; rather, they represent a careful reading of the sort that someone might do if he was interested in taking its claims seriously.
My first thought was that it’s too bad that Levitt and Dubner didn’t send a draft of their book to a careful reader like Kaiser for comments. (It’s hard to get people to comment; I routinely send draft copies of my books to zillions of friends and colleagues but usually only get a few responses. Which is understandable; people are busy.)
But then I thought, What a minute! ne person who’d I think is eminently qualified to examine the numbers in Freakonomics is . . . Levitt himself! Did he just not notice the issues that Kaiser mentioned, or was it a communication problem, that Levitt and Dubner were just too close to the material and didn’t realize that their readers might not share their knowledge base? Or perhaps they’re focused more on the concepts than the details–they like their theories and are not so concerned about the quantitative details. This can work if you’re Arnold Toynbee or Susan Sontag but maybe is riskier if part of your reputation is that of supplying your readers with interesting-but-true facts. It’s the nature of interesting-but-true facts that they’re most interesting if true, and even more interesting if they’re convincingly true.
Glad you liked the review, and that you recognized I'm not trying to debunk the book. I thought Freakonomics was pioneering in how it describes the problem of causal inference. I'm liking SuperFreaknomics less based on the 1.5 chapters I've read so far but it's still interesting to think about how other studies are designed. What was a tad disappointing about Chapter 1 was that the entire "analysis" part of the chapter consisted of computing a bunch of sample averages, and subgroup averages.
I can't say I'm particularly surprised by Superfreakonomics being sketchy on facts, even aside from the global warming bits, to be honest. Even Levitt's academic work is not particularly rigorous. Two of his seminal academic works have contained serious computer programming errors. I also find a lot of his work sloppy in that he really, really overstates the strengths of his results without considering alternative hypothesis. I mean, I'm sorry, no – analyzing NHL statistics does not give you an substantial insight into the economic-rational models of crime. The sloppiness in Superfreakonomics is also reflected in a lot of the "new social economics" literature.
Don't get me wrong, some of his stuff is quite good, but he's not consistent.
I always thought (still do!) that before the 1900s or maybe the mid-1800s, lots of women died in childbirth (like, maybe 3% or 5% of women — something like 1% per child). So I would have been pretty sure the life expectancy for women was shorter than that for men. It's interesting to hear that that's (apparently) not the case. Hmm, come to think of it, even though 3% or 5% is a big number in a lot of relevant ways, upon reflection it wouldn't have a huge effect on life expectancy. If 97% of women lived to age 60, and the other 3% lived to age 28, that would be an average of 59, so childbirth only cost a year of life on average. Not much of an effect after all.
Some of the issues Kaiser brings up (e.g. sample selection) may have been removed to make it more accessible to the non-stats-friendly public – i.e. make sure the book gets into the NYT best-seller list. One thing they could have done was to discuss those issues in appendices (online or at the back of the book).
Perhaps the most reliable data on human mortality comes from the Human Mortality Database. As you can see, females have, on average, lived longer than males since at least the 18th century.
Source: http://www.mortality.org
Sweden, Life expectancy at birth (cohort, 1×1) Last modified: 31-Oct-2008, MPv5 (May07)
Year Female Male Total
1751 35.81 32.67 34.23
1752 37.06 33.91 35.47
1753 36.67 33.68 35.17
1754 35.64 33.21 34.44
1755 35.48 32.92 34.22
1756 35.36 32.96 34.17
1757 37.36 34.14 35.76
1758 37.34 33.94 35.63
1759 38.31 34.67 36.49
1760 36.45 33.47 34.97
1761 35.80 32.56 34.19
1762 34.90 31.88 33.40
1763 36.22 32.72 34.47
1764 36.50 32.81 34.65
1765 36.72 32.96 34.83
1766 36.28 32.71 34.50
1767 34.14 30.92 32.53
1768 34.99 32.06 33.54
1769 35.68 32.97 34.35
1770 36.09 33.04 34.58
1771 34.91 31.76 33.35
1772 33.94 30.90 32.43
1773 39.28 36.18 37.75
1774 36.86 33.71 35.30
1775 37.65 34.28 35.97
1776 38.43 34.69 36.56
1777 36.65 33.15 34.90
1778 36.50 32.92 34.71
1779 37.51 33.75 35.63
1780 37.91 34.50 36.22
1781 37.83 34.27 36.06
1782 37.55 33.86 35.71
1783 38.49 34.89 36.70
1784 38.59 35.35 37.00
1785 39.61 36.01 37.84
1786 39.02 35.40 37.23
1787 39.87 36.10 38.00
1788 38.12 34.30 36.22
1789 38.82 34.97 36.91
1790 40.24 36.29 38.27
1791 40.59 36.31 38.43
1792 39.64 35.12 37.35
1793 40.51 36.51 38.50
1794 39.47 35.53 37.49
1795 40.05 35.93 37.96
1796 39.57 35.64 37.58
1797 39.98 36.14 38.04
1798 39.72 35.79 37.73
1799 38.55 34.55 36.52
1800 39.90 35.87 37.85
1801 41.25 36.63 38.90
1802 41.68 36.91 39.25
1803 40.82 36.22 38.48
1804 40.42 36.16 38.26
1805 39.94 35.80 37.84
1806 39.97 36.17 38.05
1807 40.16 35.61 37.85
1808 38.40 34.17 36.25
1809 41.38 37.13 39.24
1810 41.30 36.97 39.10
1811 41.92 37.88 39.88
1812 42.45 38.10 40.25
1813 43.70 39.34 41.49
1814 44.42 40.48 42.43
1815 44.09 39.93 41.99
1816 44.44 40.38 42.38
1817 45.23 41.06 43.12
1818 44.79 40.67 42.70
1819 45.09 41.28 43.16
1820 46.33 42.35 44.31
1821 46.68 42.94 44.78
1822 47.86 43.77 45.80
1823 47.69 43.84 45.74
1824 47.93 44.16 46.03
1825 46.83 43.15 44.97
1826 46.38 42.58 44.47
1827 46.58 42.84 44.69
1828 45.87 41.87 43.86
1829 46.10 42.14 44.10
1830 46.81 42.53 44.64
1831 47.26 43.20 45.20
1832 48.53 44.86 46.68
1833 47.98 43.94 45.94
1834 48.64 44.56 46.59
1835 49.09 45.11 47.09
1836 47.48 43.37 45.41
1837 46.89 43.14 45.00
1838 48.07 44.20 46.13
1839 49.18 45.04 47.10
1840 49.06 44.93 46.98
1841 48.81 44.82 46.80
1842 48.67 44.75 46.71
1843 49.12 45.31 47.21
1844 48.69 45.14 46.91
1845 47.78 44.42 46.11
1846 47.47 43.86 45.65
1847 48.29 44.67 46.47
1848 49.12 45.27 47.19
1849 48.54 44.87 46.70
1850 47.96 44.30 46.12
1851 47.11 43.73 45.41
1852 46.99 43.43 45.20
1853 48.06 44.87 46.46
1854 48.55 45.58 47.07
1855 48.31 45.09 46.69
1856 48.29 44.90 46.59
1857 48.72 45.47 47.09
1858 49.22 46.28 47.74
1859 49.49 46.62 48.05
1860 48.94 46.30 47.63
1861 48.86 45.72 47.29
1862 49.77 46.87 48.32
1863 49.98 47.28 48.63
1864 50.42 47.29 48.86
1865 50.35 47.50 48.92
1866 50.08 47.57 48.82
1867 48.89 46.38 47.64
1868 49.02 46.20 47.61
1869 50.85 48.67 49.75
1870 51.28 49.02 50.14
1871 51.42 49.27 50.34
1872 50.29 48.23 49.24
1873 49.87 47.56 48.71
1874 49.38 46.80 48.07
1875 49.64 47.44 48.52
1876 50.62 48.20 49.39
1877 51.63 49.05 50.31
1878 52.13 49.42 50.76
1879 52.95 50.37 51.64
1880 52.94 50.19 51.55
1881 53.34 50.68 51.99
1882 53.48 50.79 52.12
1883 54.22 51.21 52.70
1884 54.58 51.73 53.14
1885 54.87 52.26 53.56
1886 55.42 52.49 53.95
1887 56.09 52.93 54.50
1888 56.24 53.14 54.69
1889 56.28 52.96 54.63
1890 56.78 52.97 54.87
1891 57.32 53.41 55.36
1892 58.05 54.38 56.22
1893 58.79 54.93 56.85
1894 59.14 54.82 56.97
1895 59.32 55.09 57.19
1896 59.62 55.37 57.47
1897 60.18 55.94 58.04
1898 60.26 55.67 57.93
1899 60.47 55.75 58.09
1900 61.33 56.43 58.86
1901 62.27 57.45 59.83
1902 63.54 58.68 61.09
1903 64.27 59.23 61.72
1904 64.83 59.82 62.29
1905 65.38 60.04 62.69
1906 65.88 60.50 63.14
1907 66.52 61.20 63.82
1908 66.53 61.22 63.83
1909 67.53 62.22 64.82
1910 67.70 62.07 64.81
1911 68.32 62.66 65.42
1912 68.76 63.05 65.84
1913 69.00 63.24 66.05
1914 68.84 62.73 65.72
1915 69.51 63.35 66.37
1916 70.44 64.27 67.28