Some concerns about the recent Chetty et al. study on social networks and economic inequality, and what to do next?

I happened to receive two different emails regarding a recently published research paper.

Dale Lehman writes:

Chetty et al. (and it is a long et al. list) have several publications about social and economic capital (see here for one such paper, and here for the website from which the data can also be accessed). In the paper above, the data is described as:

We focus on Facebook users with the following attributes: aged between 25 and 44 years who reside in the United States; active on the Facebook platform at least once in the previous 30 days; have at least 100 US-based Facebook friends; and have a non-missing residential ZIP code. We focus on the 25–44-year age range because its Facebook usage rate is greater than 80% (ref. 37). On the basis of comparisons to nationally representative surveys and other supplementary analyses, our Facebook analysis sample is reasonably representative of the national population.

They proceed to measure social and economic connectedness across counties, zip codes, and for graduates of colleges and high schools. The data is massive as is the effort to make sense out of it. In many respects it is an ambitious undertaking and one worthy of many kudos.

But I [Lehman] do have a question. Given their inclusion criteria, I wonder about selection bias when comparing counties, zip codes, colleges, or high schools. I would expect that the fraction of Facebook users – even in the targeted age group – that are included will vary across these segments. For example, one college may have many more of its graduates who have that number of Facebook friends and have used Facebook in the prior 30 days compared with a second college. Suppose the economic connectedness from the first college is greater than from the second college. But since the first college has a larger proportion of relatively inactive Facebook users, is it fair to describe college 1 as having greater connectedness?

It seems to me that the selection criteria make the comparisons potentially misleading. It might be accurate to say that the regular users of Facebook from college 1 are more connected than those from college 2, but this may not mean that the graduates from college 1 are more connected than the graduates from college 2. I haven’t been able to find anything in their documentation to address the possible selection bias and I haven’t found anything that mentions how the proportion of Facebook accounts that meet their criteria varies across these segments. Shouldn’t that be addressed?

That’s an interesting point. Perhaps one way to address it would be to preprocess the data by estimating a propensity to use facebook and then using this propensity as a poststratification variable in the analysis. I’m not sure. Lehman makes a convincing case that this is a concern when comparing different groups; that said, it’s the kind of selection problem we have all the time, and typically ignore, with survey data.

Richard Alba writes in with a completely different concern:

You may be aware of the recent research, published in Nature by the economist Raj Chetty and colleagues, purporting to show that social capital in the form of early-life ties to high-status friends provides a powerful pathway to upward mobility for low-status individuals. It has received a lot of attention, from The New York Times, Brookings, and no doubt other places I am not aware of.

In my view, they failed to show anything new. We have known since the 1950s that social capital has a role in mobility, but the evidence they develop about its great power is not convincing, in part because they fail to take into account how their measure of social capital, the predictor, is contaminated by the correlates and consequences of mobility, the outcome.

This research has been greeted in some media as a recipe for the secret sauce of mobility, and one of their articles in Nature (there are two published simultaneously) is concerned with how to increase social capital. In other words, the research is likely to give rise to policy proposals. I think it is important then to inform Americans about its unacknowledged limitations.

I sent my critique to Nature, and it was rejected because, in their view, it did not sufficiently challenge the articles’ conclusions. I find that ridiculous.

I have no idea how Nature decides what critiques to publish, and I have not read the Chetty et al. articles so I can’t comment on theme either, but I can share Alba’s critique. Here it is:

While the pioneering big-data research of Raj Chetty and his colleagues is transforming the long-standing stream of research into social mobility, their findings should not be exempt from critique.

Consider in this light the recent pair of articles in Nature, in which they claim to have demonstrated a powerful causal connection between early-life social capital and upward income mobility for individuals growing up in low-income families. According to one paper’s abstract, “the share of high-SES friends among individuals with low-SES—which we term economic connectedness—is among the strongest predictors of upward income mobility identified to date.”

But there are good reasons to doubt that this causal connection is as powerful as the authors claim. At a minimum, the social capital-mobility statistical relationship is significantly overstated.

This is not to deny a role for social capital in determining adult socioeconomic position. That has been well established for decades. As early as the 1950s, the Wisconsin mobility studies focused in part on what the researchers called “interpersonal influence,” measured partly in terms of high-school friends, an operationalization close to the idea in the Chetty et al. article. More generally, social capital is indisputably connected to labor-market position for many individuals because of the role social networks play in disseminating job information.

But these insights are not the same as saying that economic connectedness, i.e., cross-class ties, is the secret sauce in lifting individuals out of low-income situations. To understand why the articles’ evidence fails to demonstrate this, it is important to pay close attention to how the data and analysis are constructed. Many casual readers, who glance at the statements like the one above or read the journalistic accounts of the research (such as the August 1 article in The New York Times), will take away the impression that the researchers have established an individual-level relationship—that they have proven that individuals from low-SES families who have early-life cross-class relationships are much more likely to experience upward mobility. But, in fact, they have not.

Because of limitations in their data, their analysis is based on the aggregated characteristics of areas—counties and zip codes in this case—not individuals. This is made necessary because they cannot directly link the individuals in their main two sources of data—contemporary Facebook friendships and previous estimates by the team of upward income mobility from census and income-tax data. Hence, the fundamental relationship they demonstrate is better stated as: the level of social mobility is much higher in places with many cross-class friendships. The correlation, the basis of their analysis, is quite strong, both at the county level (.65) and at the zip-code level (.69).

Inferring that this evidence demonstrates a powerful causal mechanism linking social capital to the upward mobility of individuals runs headlong into a major problem: the black box of causal mechanisms at the individual level that can lie behind such an ecological correlation, where moreover both variables are measured for roughly the same time point. The temptation may be to think that the correlation reflects mainly, or only, the individual-level relationship between social capital and mobility as stated above. However, the magnitude of an area-based correlation may be deceptive about the strength of the correlation at the individual level. Ever since a classic 1950 article by W. S. Robinson, it has been known that ecological correlations can exaggerate the strength of the individual-level relationship. Sometimes the difference between the two is very large, and in the case of the Chetty et al. analysis it appears impossible given the data they possess to estimate the bias involved with any precision, because Robinson’s mathematics indicates that the individual-level correlations within area units are necessary to the calculation. Chetty et al. cannot calculate them.

A second aspect of the inferential problem lies in the entanglement in the social-capital measure of variables that are consequences or correlates of social mobility itself, confounding cause and effect. This risk is heightened because the Facebook friendships are measured in the present, not prior to the mobility. Chetty et al. are aware of this as a potential issue. In considering threats to the validity of their conclusion, they refer to the possibility of “reverse causality.” What they have in mind derives from an important insight about mobility—mobile individuals are leaving one social context for another. Therefore, they are also leaving behind some individuals, such as some siblings, cousins, and childhood buddies. These less mobile peers, who remain in low-SES situations but have in their social networks others who are now in high-SES ones, become the basis for the paper’s Facebook estimate of economic connectedness (which is defined from the perspective of low-SES adults between the ages of 25 and 44). This sort of phenomenon will be frequent in high-mobility places, but it is a consequence of mobility, not a cause. Yet it almost certainly contributes to the key correlation—between economic connectedness and social mobility—in the way the paper measures it.

Chetty et al. try to answer this concern with correlations estimated from high-school friendships, arguing that the timing purges this measure of mobility’s impact on friendships. The Facebook-based version of this correlation is noticeably weaker than the correlations that the paper emphasizes. In any event, demonstrating a correlation between teen-age economic connectedness and high mobility does not remove the confounding influence of social mobility from the latter correlations, on which the paper’s argument depends. And in the case of high-school friendships, too, the black-box nature of the causality behind the correlation leaves open the possibility of mechanisms aside from social capital.

This can be seen if we consider the upward mobility of the children of immigrants, surely a prominent part today of the mobility picture in many high-mobility places. Recently, the economists Ran Abramitzky and Leah Boustan have reminded us in their book Streets of Gold that, today as in the past, the children of immigrants, the second generation, leap on average far above their parents in any income ranking. Many of these children are raised in ambitious families, where as Abramitzky and Boustan put it, immigrants typically are “under-placed” in income terms relative to their abilities. Many immigrant parents encourage their children to take advantage of opportunities for educational advancement, such as specialized high schools or advanced-placement high-school classes, likely to bring them into contact with peers from more advantaged families. This can create social capital that boosts the social mobility of the second generation, but a large part of any effect on mobility is surely attributable to family-instilled ambition and to educational attainment substantially higher than one would predict from parental status. The increased social capital is to a significant extent a correlate of on-going mobility.

In sum, there is without doubt a causal linkage between social capital and mobility. But the Chetty et al. analysis overstates its strength, possibly by a large margin. To twist the old saw about correlation and causation, correlation in this case isn’t only causation.

I [Alba] believe that a critique is especially important in this case because the findings in the Chetty et al. paper create an obvious temptation for the formulation of social policy. Indeed, in their second paper in Nature, the authors make suggestions in this direction. But before we commit ourselves to new anti-poverty policies based on these findings, we need a more certain gauge of the potential effectiveness of social capital than the current analysis can give us.

I get what Alba is saying about the critique not strongly challenging the article’s conclusions. He’s not saying that Chetty et al. are wrong; it’s more that he’s saying there are a lot of unanswered questions here—a position I’m sure Chetty et al. would themselves agree with!

A possible way forward?

To step back a moment—and recall that I have not tried to digest the Nature articles or the associated news coverage—I’d say that Alba is criticizing a common paradigm of social science research in which a big claim is made from a study and the study has some clear limitations, so the researchers attack the problem in some different ways in an attempt to triangulate toward a better understanding.

There are two immediate reactions I’d like to avoid. The first is to say that the data aren’t perfect, the study isn’t perfect, so we just have to give up and say we’ve learned nothing. On the other direction is the unpalatable response that all studies are flawed so we shouldn’t criticize this one in particular.

Fortunately, nobody is suggesting either of these reactions. From one direction, critics such as Lehman and Alba are pointing out concerns but they’re not saying the conclusions of the Chetty et al. study are all wrong of that the study is useless; from the other, news reports do present qualifiers and they’re not implying that these results are a sure thing.

What we’d like here is a middle way—not just a rhetorical middle way (“This research, like all social science, has weaknesses and threats to validity, hence the topic should continue to be studied by others”) but a procedural middle way, a way to address the concerns, in particular to get some estimates of the biases in the conclusions resulting from various problems with the data.

Our default response is to say the data should be analyzed better: do a propensity analysis to address Lehman’s concern about who’s on facebook, and do some sort of multilevel model integrating individual and zipcode-level data to address Alba’s concern about aggregation. And this would all be fine, but it takes a lot of work—and Chetty et al. already did a lot of work, triangulating toward their conclusion from different directions. There’s always more analysis that could be done.

Maybe the problem with the triangulation approach is not the triangulation itself but rather the way it can be set up with a central analysis making a conclusion, and then lots of little studies (“robustness checks,” etc.) designed to support the main conclusion. What if the other studies were set up to estimate biases, with the goal not of building confidence in the big number but rather of getting a better, more realistic, estimate.

With this in mind, I’m thinking that a logical next step would be to construct a simulation study to get a sense of the biases arising from the issues raised by Lehman and Alba. We can’t easily gather the data required to know what these biases are, but it does seem like it should be possible to simulate a world in which different sorts of people are more or less likely to be on facebook, and in which there are local patterns of connectedness that are not simply what you’d get by averaging within zipcodes.

I’m not saying this would be easy—the simulation would have to make all sorts of assumptions about how these factors vary, and the variation would need to depend on relevant socioeconomic variables—but right now it seems to me to be a natural next step in the research.

One more thing

Above I stressed the importance and challenge of finding a middle ground between (1) saying the study’s flaws make it completely useless and (2) saying the study represents standard practice so we should believe it.

Sometimes, though, response #1 is appropriate. For example, the study of beauty and sex ratio or the study of ovulation and voting or the study claiming that losing an election for governor lops 5 to 10 years off your life—I think those really are useless (except as cautionary tales, lessons of research practices to avoid). How can I say this? Because those studies are just soooo noisy compared to any realistic effect size. There’s just no there there. Researchers can fool themselves because the think that if they have hundreds or thousands of data points, that they’re cool, and that if they have statistical significance, they’ve discovered something. We’ve talked about this attitude before, and I’ll talk about again; I just wanted to emphasize here that it doesn’t always make sense to take the middle way. Or, to put it another way, sometimes the appropriate middle way is very close to one of the extreme positions.

20 thoughts on “Some concerns about the recent Chetty et al. study on social networks and economic inequality, and what to do next?

  1. Whoa. Chetty et. al discover the Sneetches! :) Until now it was a secret to us secretive Seussians that hooking on with wealthy trend-setters could move you up in the world. Dagnabit! Chetty et. al. have exposed our secret!

    OK, I admit, this isn’t an exact replication of The Sneetches. But The Sneetches illustrates the same principle implicitly: the plain bellied Sneetches want to imitate the Star-bellied Sneetches because Imitating the Star-Bellied Sneetches is the door-opener to gain entry to the SBS social group.

    I love this description of the story from Wikipedia:

    “An entrepreneur (and scammer) named Sylvester McMonkey McBean (calling himself the Fix-It-Up Chappie) appears and offers the Sneetches without stars the chance to get them with his Star-On machine, for three dollars. The treatment is instantly popular…”

    Howard Schultz = “Sylvester McMonkey McBean”
    $5 Starbucks Coffee = Sneetch Star

    Just like in the Sneetch story, however, the Starbucks cup has lost its street cred among the wealthy as the company has gone down market. I don’t know what the new symbol is. I should visit my sister. She’s a Sneetch Chaser from way back.

      • That’s hilarious!

        But FWIF, I have no particular opinion SBUX or Howard Shulz. I’m a Buffettarian in that respect: happy to own the stock but not likely to dig a fiver out of my wallet for a cup of coffee – unless of course I need to flash some social cred. :) Just casting the characters in their roles.

        The “Fix-It-Up Chappie” – I love that! – could just as well be any politician, including Biden and Trump in equal measures; or any of the Nudgistas in politics or social science.

  2. Alba’s critique leaves me pessimistic about the program you sketch in the “A possible way forward?” section.

    Alba seems to suggest that we have long known about the phenomenon under discussion, in which case the value of the study could at best be (a) further confirmatory evidence, and (b) quantitative estimates of the effect size. But for the reasons explained by Alba, the latter estimate is questionable.

    We could try to improve or refine the estimate or get a better sense of uncertainty through the plan you suggest. But to what end? Alba says it is important to get a better estimate before forging ahead with interventions guided by this observation. But if our ultimate goal is policy changes that exploit this phenomenon, like the ones mentioned at the end of the second Nature paper, then I’m not sure how much even a refined estimate is actually going to help inform these decisions. For example, the three suggested policy changes discussed are:

    1) Eliminating “tracking” in schools (i.e. not separating classes based on ability)
    2) Designing buildings/spaces to promote interaction
    3) Creating programs for interaction across SES lines

    Does knowing whether the “friend bias” effect magnitude is X or 2*X actually help us weigh the trade-offs involved in any of these policy changes? I doubt it. There are so many layers of issues and intermediating effects in between this raw estimate of “friend bias” and the ultimate outcome of such programs. As a result, if one is ultimately interested in implementing such interventions, it seems you’d be quite a bit better off directly designing experiments to test them, rather than trying to first improve estimates from Chetty et al.’s studies. I suppose the estimates from the studies discussed here could help inform priors for these future policy experiments, and maybe they’re already good enough for that purpose?

    • It is like the opening lines of a dystopian sci-fi:

      “In 2022, NHST lead to humans considering cronyism/nepotism as a good thing rather than age-old nuisance to a functioning society. Efforts were taken to reduce meritocratic incentives and take giving all the best jobs to family and friends to the extreme logical conclusion.”

      • I agree it’s remarkable how cronynism/nepotism is being framed as something positive. For me however it has little to do with NHST and more about the value system at play. Would the message be any different if they used Bayesian methods?

        As to what this value system is, I have my guesses but it feels unfortunate that the social sciences are not up to the task of addressing these issues.

  3. Alba is plainly correct. Not only does the article not show anything new, it does not show anything. It’s is nothing but tabloidy “big-data” research making huge claims based on vacuous data (what does a Facebook friendship actually mean sociologically?) with ridiculous methodology (as Alba points out, “their analysis is based on the aggregated characteristics of areas—counties and zip codes in this case—not individuals”)

    • Right, we know how to actually collect meaningful or at least better social network data and that understanding strength, direction and qualities of the ties is important. Facebook data is really not the way to go, and then aggregation by area really is going to hide these micro level impacts.

    • Mark said, “what does a Facebook friendship actually mean sociologically?”

      Good question. Perhaps before using Facebook data to try to study some sociological question, the researchers need to think about how people become (or do not become) Facebook friends. For example, I first got on Facebook in response to an “invitation to join”, that one of my cousins sent out to all her cousins. After reading about how the platform works, I decided that I would abide by two rules: 1) I wouldn’t initiate any friend requests; 2) I would only accept friend requests from relatives or very old family friends who were almost like relatives. So my Facebook friends are almost all people whom I’ve know since childhood, plus their spouses and offspring. I wonder how many people using Facebook have established some kind of guidelines or limitations in becoming Facebook friends with someone.

  4. We are at the goldilocks zone of academic research where

    1. It is surprisingly easy to create/find legitimate looking academic papers as evidence of your belief, whatever that may be.

    2. When confronted with papers that contradict our beliefs, it is also easy to criticize its limitation. (p-hacking, measurement error, forking paths, etc.)

    It sucks for people who still hold out on the idea that the goal of academic research is to find the “truth,” but works really well for everybody else.

    • The problem is they perform some steps of science but not the others.

      They collect some data and explore it, finding various correlations. That’s great.

      Then they are supposed to explain why those correlations have the values they do. I don’t see any theoretical curve in the paper, just descriptive lines fit to the data. So there doesn’t appear to be any quantitative theory involved here, which means there are many explanations that will fit the data just as well as their own.

      Then they are supposed to deduce a prediction from that theory and compare it to new data, which they also don’t do. It could also be useful to only have an empirical model that makes accurate predictions, in which case we would expect to see a hold-out dataset so we can see out-of-sample performance. That also appears to be missing, or at least this (very important point!) is not made evident in their description.

      Then they draw conclusions about the process that generated the data. That is great too, but unfortunately they failed to develop their theory into one capable of generating quantitative predictions… or even show they have discovered a stable relationship worth theorizing about to begin with.

      There is no reason to think this half-baked process will convince anyone who doesn’t believe their conclusions to begin with. Basically they did the first and last steps of science, but skipped the important part in the middle that makes it work. And this is *very* common.

  5. It is interesting how these comments are so dismissive of this research. Chetty is a superstar in economics, probably headed for a Nobel. But the limited comments seem quick to dismiss his work as redundant and/or flawed. I’m not saying I disagree, but I want to make an observation. I’ve seen this happen before on this blog – for example with psychology (particularly some sub-disciplinary fields like social psychology) and epidemiology, and data science…. The observation I have is that often smart people seem willing to dismiss research in other fields very quickly as uninteresting, misguided, or even wrong.

    I think this is a reflection of the serious over-specialization that exits in academia. And, the journal and grant process feeds this. Perhaps work should not be “peer-reviewed” but instead reviewed by experts in different fields. It appears that much research takes place that would not pass muster that way. And, I think much research that takes place probably is not furthering any important understanding about the world. Could it be that our research paradigms are misaligned with senses of importance? They have become so internally focused that they lose sight of any bigger picture. And then the irony is that this research is then picked up by the media and hyped as being more relevant than it is.

    • Dale;

      See the final section (titled “One more thing”) in the above post. I think the Chetty et al. research has the potential to be valuable, and also it’s useful to think about its limitations. But there is other research out there, some of it by well-credentialed scientists and appearing in well-credentialed journals, that I think really is junk. I would not say that I, or other critical scientists, have been “quick to dismiss” that junk; rather, it took years for us to get a sense of how useless it can be.

      • “He’s not saying that Chetty et al. are wrong; it’s more that he’s saying there are a lot of unanswered questions here”

        “In sum, there is without doubt a causal linkage between social capital and mobility…” (Which was discovered by social science “As early as the 1950s”). “But the Chetty et al. analysis overstates its strength, possibly by a large margin.”

    • > I think this is a reflection of the serious over-specialization that exits in academia. And, the journal and grant process feeds this. Perhaps work should not be “peer-reviewed” but instead reviewed by experts in different fields. It appears that much research takes place that would not pass muster that way.

      My reaction is almost the opposite, especially given that the paper under discussion appeared in Nature. In my area (which is perhaps not representative), work that appears in Nature is almost always derided for being flashy, “pop” work, with results that superficially sound appealing to cross-disciplinary experts, but which would not pass the bar for rigor in the top area-specific journals. But I suppose this is highly area specific: if a field has pervasive methodological issues, then over-specialized journals are probably likely to let bad practices slide.

  6. Generally speaking I like Chetty’s work, he and his group are creative in terms of finding interesting ways to look at topics that economists often don’t think about. (And I can self interestedly say that his work on which colleges have the biggest impacts on their students found that my institution was close to the top, so I think that work is really great.) But, this is an example where economists discover something that has been studied and theorized in other disciplines for decades. The idea that social capital alone — that is the amount of connection to other people — is useful was long ago rethought into the idea that it matters what kind of structures the capital represents. So, yes, “bridging” social capital has more payoff than social capital alone because bridging social capital gives you access to more resources. That means a closely knit homogeneous enclave/community where everyone helps each other is better than not knowing anyone, but a less dense enclave combined with connections to people (and hence resources) outside it is even better. In other words if you have 10 friends and they all are friends with each other, because their networks overlap so much, if you are looking for a job they probably won’t help as much as having 10 friends some of whom are not friends with each other.

    https://documents1.worldbank.org/curated/ru/515261468740392133/pdf/281100PAPER0Measuring0social0capital.pdf
    Social capital is about trust and whether people will help you in times of trouble or whether you can organize collectively to address an issue. I’m not sure that their data captures this, but more important, it is that this kind of data needs to be analyzed in a way that reflects the inherently micro level concepts.

    So overall, I think this study is in line with what others have found, but the measurement design has problems.

    Also, I think this study fits into Chetty’s larger idea of “move to opportunity” and zip code as destiny. Social capital can explain why moving to a higher status area can be helpful and why even with personal resources and human capital living in a given place can still hold you back. But my response to that is always, so is everyone supposed to just move out of the The Bronx? It doesn’t really seem helpful.

Leave a Reply

Your email address will not be published. Required fields are marked *