GIGO

Lee Wilkinson writes:

In the latest issue of Harvard Magazine (http://www.harvardmagazine.com/2015/12/cambridge-02138), a letter writer (David W. Pittelli) comments under the section “Social Progress Index”:

We are informed by Harvard Magazine (November-December 2015, page 15) that the country with the best “Health and Wellness” (“Do people live long and healthy lives?”) is Peru, while the United States ranks a dismal sixty-eighth in the world.

This seemed unlikely to me, and so I went to www.socialprogressimperative.org to see how Social Progress Imperative (SPI) arrived at its statistical claims.

The broadest statistic making up the Health and Wellness (HW) rating is Life Expectancy. From the figures, we see that the United States has a life expectancy a full four years longer than that of Peru (78.7 vs. 74.5 years). So how does SPI come to a figure that puts Peru at best in the world? They add other figures related to death, such as “Premature deaths from non-communicable diseases,” which are somewhat higher in the United States than in Peru. But why should we double-count a death from a noncommunicable disease like a heart attack or diabetes, which strikes mostly in advanced nations, but ignore a death from a communicable disease, most of which are more common in poorer nations like Peru?

The writer’s letter is longer in the printed version of the magazine, where he goes on to point out that several composite statistics for the countries aggregate over individuals that are in many cases counted twice or more. (See here, which has the juicy bit that Saudi Arabia gets credit for a secondary school enrollment rate that is above 110%!)

In the printed version of the magazine, Michael Porter (Harvard Business School’s TED-talk professor) comments on this letter, specifically refuting Pittelli’s claim:

There is also no double counting, as one reader suggested. Our principal component methodology is specifically designed to minimize or eliminate it.

The relevant SPI methodology document describing how the components were computed is here.

Aside from the apparent confusion the SPI authors have over the difference between Factor Analysis and Principal Components (perhaps because their “authoritative” source is Manly, Bryan F. J.
Multivariate Statistical Methods: A Primer. CRC Press), there are two immediate questions:

1. Porter’s claim that there is no double counting has nothing to do with Principal Components methodology. As Pittelli says, the aggregation of various statistical measures was done prior to the computation of the components. Including measures that involve more than one individual makes the aggregate measures ambiguous. Interpreting the resulting principal components is subject to this artifactual bias.

2. Inferences on principal components (and other classical multivariate statistics) depend on a covariance matrix that does not include structural dependencies. If two or more measures are dependent partly due to a shared nonrandom component, the standard Principal Components model is a misrepresentation of the variation in the data.

I may be misinterpreting the survey methodology, and I may be quibbling, but this apparently influential survey appears to merit some deeper investigation.

I responded: this sort of thing is just crap, something like the notorious Places Rated Almanac or US News rankings. I wrote about something similar several years ago; see here and here.

But as long as the “garbage out” gets media attention, there will always be somebody willing to supply the “garbage in.”

P.S. I googled Ted-talk professor Michael Porter who was quoted above, and I came across this, entitled “The case for letting business solve social problems”:

I’m not a social problem guy. I’m a guy that works with business, helps business make money. God forbid.

OK, so far so good. I help businesses make money too, that’s part of what I do.

But then a few minutes later:

I’m a business school professor, but I’ve actually founded, I think, now, four nonprofits. Whenever I got interested and became aware of a societal problem, that was what I did, form a nonprofit.

And then a bit later:

I’ve also, over the years, worked more and more on social issues. I’ve worked on healthcare, the environment, economic development, reducing poverty, and as I worked more and more in the social field, I started seeing something that had a profound impact on me and my whole life, in a way.

All I can say here is . . . c’mon, make up your mind, dude. There’s nothing wrong with being “not a social problem guy”—not everyone can work on social problems. But then don’t pop up 3 minutes later to brag about how you solve social problems!

This particular speech seems to be approx 15 minutes of pure happy talk about how doing good is actually more profitable than doing evil. I wonder if someone could lock this guy up in the room with Gary Loveman or, even better, those guys who keep filling my inbox with spam.

I see no direct connection with the bogus statistics that Porter is touting and his doing-well-by-doing-good shtick, except perhaps for a general willingness to say whatever can make his point, without caring whether it makes any sense or is even internally consistent.

I’m not saying Porter has any particular political bias: politically, he seems quite centrist. When he says that all wealth is actually created by business, that sounds conservative, but he sounds liberal when he talks about gay rights or about reducing pollution being good business. I looked up his political contributions on the FEC site and I found a Michael Porter from the Harvard Business School who’s given money to Scott Brown, Steve Pagliuca, Nancy Johnson, Jeff Bingaman, William Binnie, James Ogonowki, and the Massachusetts Republican Party. This is the signature of a political moderate. So it’s not like I’m saying Porter’s statistical shenanigans are serving some extreme agenda; I just think he’s sloppy and doesn’t really care about the facts, he’s just throwing numbers around to illustrate whatever point he happens to be making. He probably gives an excellent Ted talk.

40 thoughts on “GIGO

  1. If you create your “index” by doing I = 2*LifeExpectancy + 1*AvgLifeYearsLivedNotInPoverty

    You are actually “triple counting” some of the years, first off, you’ve got a factor of 2 in front of the life expectancy… so that doubles those years, and then you’ve got the years where you weren’t in poverty counted again…

    but, I’m sorry, I don’t see anything “wrong” with this. Double counting things just means you value them more in your metric. You’re expressing some kind of value system here, and it’s totally appropriate to value some things more than others.

    Am I missing something? (except for the fact that hey, all indexes should be dimensionless… so divide those terms by maximum credible life expectancy, say 105 years or something)

      • I think it went beyond that: “But why should we double-count a death from a noncommunicable disease like a heart attack or diabetes, which strikes mostly in advanced nations, but ignore a death from a communicable disease, most of which are more common in poorer nations like Peru?”

        Sure, this is a good question (why) but it’s not out of the question to say that we should weight some kinds of things higher and other kinds of things lower. “Double counting” is another way of saying “weighting this thing higher”. It’s not *inherently* wrong but it does need justification.

        as for 110% attendence, I think that’s just a data error, not a questionable structural choice.

        When it comes to the PCA analysis, classical inference on PCA components could depend on the phase of the moon for all I care, frequentist statistics are probably the wrong way to assess PCA components. The proper way to understand PCA is as a rotation of the axes of a high dimensional space so that certain axes correspond to the directions along which there is a lot of variation in your data. The typical next step is to ignore the variation along other axes as being “too small to matter” and then analyze just the N “largest” dimensions.

        Note, that in order for that to be a good idea, you NEED to normalize your data to a meaningful scale. If I measure two things, say height in nanometers, and dollars in your pocket in thousands… and I don’t standardize those to meaningful measurements (ie. by dividing by say 2×10^9 nanometers and 0.1 thousand $) then my PCA will be highly skewed towards the “highly variable” nanometer measurement, its measurements vary over a scale of 10^9!!

        There is no reason to believe that a Bayesian analysis of a PCA decomposition can’t be done if there are structural issues within the covariance matrix. This poses no problem whatsoever so long as you have such a model.

        • Daniel:

          I agree with you that indexes can be useful even when the data that go into the indexes are not perfect. But I think this particular index has serious problems—I agree with Pittelli that if Peru is #1 and the U.S. is #68 in the “long and healthy lives” measure, that at the very least we should figure out where in the index this is coming from. Yes we can learn from surprises—but in this case what we learn from the surprise is that the index has big problems. And in any case you want to fix the bad data such as 110% school attendance rates.

          To put it another way, using your terms: I don’t think the creators of this index had a reasonable model. It looks like they had some data which they didn’t check very well, and they had a statistical procedure which they didn’t understand very well, then they piped the data through the procedure and didn’t look seriously at the results. Then when these results were criticized in a reasonable way, the creators of the index did not use this as an opportunity to improve their method. Bad practice all the way through—especially for someone like this Michael Porter guy who clearly has the resources to do better.

        • I agree with all of that. I’m definitely not defending this index, I’m more arguing that the core problem with the index is at the modeling level, not the implementation level. “Double counting” sounds like an accident that should be fixed and then everything would be ok…. it’s not the real issue here. The real issue is exactly what you mention.

        • For what little I understand, it is quite usual to normalize each dimension on standard deviation (or some other measure of dispersion) while doing PCA.

        • Yes, but this is equivalent to the statement that “observed variation in this data set corresponds to importance”. It’s a little like confusing statistical significance and practical significance.

        • Depends on your problem. If you are studying the main predictors of student’s success on a test and throw in a bunch of observables, you have to care to have a representative sample, but not much about “importance”. On the other hand, if you want to use this finding to give students some advice, you have to care about how practical it is that they will follow it (it’s not very practical to ask them to change their socioeconomic status, to say nothing about ethnicity).

        • Yes, it definitely depends on your problem. I think that’s my point. in some problems 1 observed standard deviation is an important degree of change, and the variable is more or less randomly assigned (such as parents income in a student analysis). In others, not so much. For example, suppose one of your variables is under your experimental control. Let’s say it’s days since treatment of a plant with a fertilizer, which has also say some measurement error, you wouldn’t necessarily want to divide this by the observed sd automatically.

          but, I do think this is almost an automatic transform in some software, mistakenly. Or this technique is suggested as good advice to people who are end-users and the point is not explained properly, it can become a rote thing without thinking.

        • Re the 110%. According to a quote in that blog that Andrew links to, it is not a data error but a function of how full enrolment is calculated by the World Bank. I am too lazy to go back and find it, but essentially the figure seems to imply that there are over or under age students enrolled in the Saudi school system.

          My guess, and it is only a guess, would be that, given the large number of foreign workers in the Kingdom, we are seeing some foreign children lagging in the educational system due to language or poor preparation before arriving in the Kingdom.

          However I know nothing about the Saudi school system except I would not have a lot of faith in the high school graduation marks.

        • Hi, I am the letter writer referenced in the initial post.

          The 118% and 110% attendance rates in Saudi Arabia aren’t bad data, they reflect that a lot of people in Saudi Arabia are below grade level, such that the number of people in, say, middle school, is higher than the population that is middle-school-aged. In the creation of the index, all the numbers higher than 100% are limited to 100%, as Professor Porter noted in his reply to my letter. As the Methodology Report puts it, “the indicators which measure gross school enrollment have been capped at 100 percent to prevent countries from being rewarded for students repeating grades.” However, that still puts the Saudi number (now 100%) as higher than the 98% in the United States, so it still “rewards” Saudi Arabia for its educational failures.

          I don’t think the PCA makes much difference; the 5 individual indicators making up each of the components is averaged. While it is a weighted average, the weightings are close to even; for “Access to Basic Knowledge” the weight of each of the 5 indicators is listed in the Methodology Report as between 0.18 and 0.21.

    • I think the argument is that it is a value system that is difficult to justify. It also seems to ignore the work already done to develop reasonable summary measures of health attainment. Sure you can posit an index, but why should we care? How are the different domains meant to be interpreted, noting that while health has its own domain, maternal mortality, infant mortality, communicable disease mortality, deaths to household air pollution, and homicide deaths each show up in a different category?

      • Sure, I totally agree with what you’re saying here, but that’s a different question than whether you’re “double counting”. It’s not that somehow “double counting” is inherently bad, it’s that you need to justify why you set up your scheme the way you did, and be aware that including thing X may be equivalent to giving bigger weight to thing Y or whatever.

      • I think you can dispute whether it was calculated accurately, ie. you might be using bad information in the index, But you can’t dispute the values expressed by the choices of how to create the index. You can however, have a discussion about whether it represents something that you or some particular group actually cares about.

        • We can dispute the goodness or badness or morality etc of the values, but there is no sense in which we can dispute whether they are somehow correctly calculated, there is no objective standard to determine whether index I “correctly” expresses person X’s values. That’s all I meant.

        • “Objective standard” is somewhat loaded term. It is pretty clear that person’s ability to whistle has very little bearing on their well-being even if you cannot rule it out objectively. And there is a whole profession — economics — that would vehemently disagree with the general sentiment. And then, of course, you might disagree with them.

        • An index like this expresses SOME hypothetical value system… whether that value system is actually held by *anyone* is hard to argue for or against, whether it’s the same as *your* value system is easy to dispute. I think that’s all Rahul was saying.

    • There is nothing wrong about it per se, you simply express the idea that a year of life in poverty is worth 2 utils, while a year of life out of poverty is worth 3 utils. The problem with the index described in OP is that it seems arbitrary and capricious, not mathematically deficient. We might think that death of a heart attack is a signature of years of life of poor quality (because of the disease) and use it as a proxy, but any such choice has to be justified.

  2. I only knew of Porter for his work on competitive strategy. He is very well known in the business community for it. So it would be news to me that he is involved in social problems or starting non-profits.

  3. 1. When you are a person of broad interests, you have to be careful not to think of yourself as an expert in all of them. This may be particularly hard when you have a job at the Harvard Business School, one of the 100 or more “top 10” business schools in the U.S. [of course, HBS is one of those few schools that will make everybody’s list.]

    2. Sure, these ratings are crap, but we have an odd fascination with them, and if you are trying to get attention for your nonprofit work, they are an inexpensive way to go for attention. For example, I seem to recall a LinkedIn(?) post a few years ago trying to rank “the 3 worst statistical procedures” or something like that. It generated many more responses than the average post. (I think multiple regression won/lost; I can’t find the link. Naive Bayes was up there, too.)

  4. Are people here really that unfamiliar with Porter? He is, arguably, the most famous business academic in the past 30 years. Yes, competitive strategy is his most famous work, but he has indeed branched out into many fields. He believes he has the answers to health care costs (and I find his answers less than satisfying). None of this is meant to elevate his statements – I just was shocked to see that his name aroused so little reaction. I agree with zbicyclist – this is a danger shared by many people (economists are first on my list, perhaps because I was trained as one) who believe they are experts in everything because they know something.

    • Never heard of him, I’ll admit.

      Somehow stuff like “competitive strategy” seems kinda hand-waving heavy, and ephemeral. These business school topics seem to have a short shelf life. The fads come and go.

      One decade they tell us companies ought to focus and divest every business except their core competency. The next decade “diversification” will be the thing to do.

      • No, Porter’s 1980 book “Competitive Advantage” is the real deal in that it explains clearly that the reason you are in business is not to compete but to to get some kind of monopoly advantage so you don’t have to be a perfect competitor, those losers. It’s been very influential on my thinking ever since.

    • I do remember him being thought as a strong contributor to competitive strategy when I worked with this group – Canada Can Compete!: Strategic Management of the Canadian Industrial Portfolio. Joseph R. D’Cruz, James Douglas Fleck

      But high intelligence (in one field at one point in time) is no defense against stupidity ( I have not read the post very carefully ).

    • HBS professor Michael E. Porter’s book on competitive advantage was eye-opening to a naïve economics major like myself when I was studying for my MBA in 1980-82. Your undergrad Econ 101 professor explains how a wheat farmer is a “perfect competitor,” which sounds pretty cool. When you get to B-school, however, your business strategy professor points out that you do not want to be a wheat farmer. Perfect competition is no fun at all. As Porter wrote in 1979:

      “The essence of strategy formulation is coping with competition. … In the economists’ “perfectly competitive” industry, jockeying for position is unbridled and entry to the industry is very easy. This kind of industry structure, of course, offers the worst prospect for long-run profitability. … The corporate strategist’s goal is to find a position in the industry where his or her company can best defend itself against these [competitive] forces …”

      http://takimag.com/article/better_a_crook_than_a_wasp_the_left_ditches_progressivism_steve_sailer/print#ixzz45ZyXvDsu

      • I see some of these strategies in practice & it makes me cringe: You want to block low-cost competition? Tell you pals at the FDA to raise the bar for manufacturing facility certification.

        Put such a low assay limit on some irrelevant impurity that buying a $100,000 analyser is the only way to detect it. Lobby the downstream industry association to institute third party “supplier audits”. Even better, require “social responsibility audits”.

        Require stamps on pressure vessels where the industry association handing out stamping rights is filled with your cronies.

        Strategies ad nauseum.

  5. New Zealand seems to score well on his index so perhaps I shouldn’t complain, but Michael Porter was well known in this part of the world 25 years ago when our government commissioned him to write a report titled “Upgrading New Zealand’s Competitive Advantage” (but known colloquially as the Porter Project.) Back then (perhaps not quite so much now) our government departments were in thrall of any overseas “expert” who could tell us what to do, and especially so if their philosophy happened to align with that of the government of the day.

    Anyway this critique written at the time by one of our leading political economists suggests that his presentation and analysis skills weren’t the greatest back then either – http://www.eastonbh.ac.nz/1991/06/the_porter_project/

    • Awesome!

      “Its 178 pages (plus appendices) are riddled with badly labelled graphs; portentous diagrams which, on reflection, say nothing; chummy references to “our country”, when two of the three authors are Americans; and platitudes dressed up as ‘deep and meaningful sentiments.'”

    • Howard:

      I like this quote from the review:

      Particularly galling is the book’s claim that we should improve the efficiency of government spending. The funding of this report would have been a good place to start. It must be a candidate for the lowest productivity research publication ever funded by government.

      Ouch!

  6. Speaking of life expectancy, there’s a new paper from economist Raj Chetty drawing on his huge IRS database.This time he’s trying to find out where (race-adjusted) life expectancy is longest and shortest relative to income. His results are pretty amusingly paradoxical: the poor live longest in plutocratic places lacking in affordable housing, like New York and Santa Barbara, while the affluent live longest in socially conservative, not very diverse places like Salt Lake City and Grand Rapids.

Leave a Reply to David Pittelli Cancel reply

Your email address will not be published. Required fields are marked *