Ray Fisman sent me an email with the above title and a link to the following story.
Last year this article was published in the Proceedings of the National Academy of Sciences:
A few months later the journal published this letter criticizing the above-linked paper:
A subset of the original authors then replied in the journal:
Here’s the key bit from the original article:
Cohn et al. (2019) conducted a wallet drop experiment in 40 countries to measure “civic honesty around the globe” . . . we conducted an extended replication study in China, utilizing email response and wallet recovery to assess civic honesty. We found a significantly higher level of civic honesty in China, as measured by the wallet recovery rate, than reported in the original study, while email response rates remained similar. To resolve the divergent results, we introduce a cultural dimension, individualism versus collectivism, to study civic honesty across diverse cultures . . .
And here’s the central criticism in the letter:
A key finding in YAC is city-level collectivism predicts safekeeping but not emailing, which would suggest civic honesty expresses itself differently across cultures. This result, however, is entirely due to an error in their regression specifications; once corrected, the relationship between collectivism and safekeeping disappears. YAC’s regressions include both city fixed effects (i.e., where the study was performed) and city-level rates of collectivism (i.e., degree of collectivism in a city). Including both variables leads to . . . perfect multicollinearity. . . . arbitrary changes to the model—which should have zero effect on the collectivism coefficient—alter the coefficient from significantly positive to significantly negative to no longer estimable. . . .
The reply by the original authors had two main points:
Tannenbaum’s comments about the model specification were based on an erroneous term, “city-level collectivism.” Yang used provincial-level collectivism, a higher level of aggregation, unlikely to form “perfect multicollinearity” with the city-level factors . . .
Yang acknowledged that the replication is a small-sample study with limited power. Therefore, the nonsignificant relationships identified by Tannenbaum, even with repeated simulations, cannot exclude true associations between collectivism and civic honesty. Rather than removing city-fixed effects, per Tannenbaum, the appropriate approach is to recruit more cities within provinces in future experiments. . . .
My reactions
I have not studied the substance of these studies, so I’ll just comment on the two statistical issues that arise:
1. If you try to fit a least-squares regression including indicators for groups along with a group-level predictor, you will indeed get collinearity. Actually, you’ll get collinearity if you include indicators for groups without a group-level predictor. If your model has J groups and K group-level predictors, your statistical software will typically handle this problem by dropping K of the group indicators from the regression. Tanenbaum et al. are correct that, unless this is done very carefully, it will destroy the interpretation of the coefficients of the group-level predictors.
2. In their reply, Zhang et al. say that, because their predictor, provincial-level collectivism, is at a higher level of aggregation, there won’t be perfect collinearity. They are incorrect. A predictor on a higher level of aggregation is still a group-level predictor, and it will be collinear with the group-level indicators. Instead of trying to wriggle out of it, Zhang et al. should’ve just thanked Tanenbaum et al. for pointing out their mistake.
3. The way I recommend dealing with this problem is not to drop the group indicators, but rather to keep them as varying intercepts in a multilevel model. Or, if you really don’t want to fit a multilevel model, do some analysis that adjusts for within-group correlation in the data. That “collinearity” thing is not an issue with the model at all. It’s entirely an issue with fitting the model using least squares. Switch to a multilevel model and the problem goes away.
4. In their reply, Zhang et al. describe their earlier published experiment as “a small-sample study with limited power” with “nonsignficant relationships.” That’s fine—but then they should retract this strong claim from their published paper:
Our findings reveal that the variations in collectivistic/individualistic values can explain the differences in behaviors across cultures, which has significant implications for understanding cultural differences in civic honesty.
A small-sample study with limited power and nonsignficant relationships does not “reveal” anything; nor can it have “significant implications for understanding.”
As an advocate of publishing everything, I’m the last person to suggest that lack of statistical significance should be a barrier to publication. What I don’t like is when people overstate the evidence for their claims.
Finally, regarding Fisman’s characterization of this episode as “a bizarre failure in the review process at PNAS,” my reaction is: Are you kidding??? PNAS published the notorious papers on air rage, himmicanes, ages ending in 9, nudges, . . . I’m sure I’m missing a few notorious examples. Of course they’ll publish a paper with statistical errors, if it makes strong and newsworthy claims. There’s no reason to think the reviewers were aware of these errors. The reviewers just missed the problems, that’s all. It happens all the time.
Again, none of the above addresses the substantive questions of the exchange, interesting as they are. I’m just writing about the statistical issues, about which there seems to be a lot of confusion all around.
P.S. There’s also a political dimension here. Fisman points to this article from a Chinese government source giving a highly nationalistic take on the story.
It looks like the original authors shared the data. Should be fairly straightforward to re-analyze and evaluate findings via Bayesian multilevel modeling. Will try during my lunch break! https://github.com/Science-replication-group/Unraveling_Controversies_over_Civic_Honesty_Measurement
The authors may have goofed in their original analysis, but by sharing their data they’re contributing much more than most other research groups.
Good luck with the afternoon analysis!
Soooo….the outcomes are binary, but coded as 0 and 100 so that the authors can fit linear probability models. I’ve never seen that before (in my field)! I’ll make them dichotomous and fit GLMs and see if the magnitude, direction, and precision of the trends hold.
It seems like the thrust of the narrative really only hinges on one of their three outcomes (returning the wallet), as there wasn’t much of a difference in “collectivism” on the impact of the social experiment on emails or total recovery of the wallet. I refit their models using multilevel Bayesian logistic regression (clustered by city, institution, and day). I didn’t explore random slopes. Code, data, and main plot of main finding here: https://drive.google.com/drive/folders/16tEac7sI4X4-_rAxAA5wBd4Mev-tWUqi?usp=sharing
Re-analysis shows that the “big finding” about the importance of (the proxy for) collectivism only holds for one of their three outcomes, and only just. Nothing for the CCP to prop up. Happy to share the code and results (I can’t seem to link them here, given moderation).
This cycle irks me to no end:
1. Conduct poorly conceived study
2. Get non-significant results
2a. (optional, but conventional). Torture data until you obtain non-but-nearly-significant results
3. Make confident inference on these results; fantasize far-reaching implications of this inference
4. Claim that the _main_ limitation of the study is low power, which of course shows that the most important finding of the study is that it Merits Further Investigation and thus you deserve *more* funding for landing on the margin of significance. Gods, if only they had given you more money you would have cracked the code!
5. Do NOT further investigate the same effects, you aren’t _that_ stupid.
6. Pull 7 variables out of a hat. Return to Step 1.
Repeat until tenure, TED talk, NPR interview, etc….
I love the dueling stereotypes of Chinese culture. In my personal experience, half of ethnic Chinese think that authoritarian politics works for the PRC because of collectivist Confucian culture, and the other half think that Confucian education and authoritarianism is uniquely necessary to govern the PRC because Chinese people are socialized to be conniving and selfish.
If you go by Chinese ethnic enclaves stateside, with the enormous market for stolen and bootlegged goods, medicinal scams, religious scams, cults, illegal gambling, American Chinese business owners tricking Chinese Chinese employees into coercive illegal labor conditions, the impression I get is that Chinese culture is like any other culture where there’s a well capitalized and educated class interacting with a desperate, poorly educated class. I’m told that these enclaves are more of a snapshot of coastal China from the early-mid 20th century though; perhaps things have changed.
This is a fun one–the examples feel hilariously familiar
https://cup.columbia.edu/book/the-book-of-swindles/9780231178631
Taking a closer look at the actual paper.
Yes, when someone walks up to you with their face in your face and asks about their wallet, people are more likely to give the wallet back than if they just waited to be contacted. That’s likely to be true everywhere; without repeating the “walk back and ask for their wallet” experiment in the other country’s in Cohn’s original study, this study’s stated goal of resolving the controversy is dead in the water. By looking at sub-national covariates, I suppose it can be at least suggestive of a further cross-national comparison, but the covariates:
Oh good lord! Linear probability models on proxies on proxies
other half think that Confucian education and authoritarianism is uniquely necessary to govern the PRC because Chinese people are socialized to be conniving and selfish.
Interesting, because that hasn’t been my experience at all.
The Chinese people I’ve worked with and talked to (quite a few) almost uniformly see a practical, societal benefit to “Confucian” principles and are generally bemused as to why individualism fetishizing Americans often have a difficult time seeing their value.
“Conniving and selfish” actually strikes me as a very American-centric viewpoint of “Chinese people.” It seems necessarily rooted in an individualistic paradigm all the way down.
Shit. Reformatting:
…the other half think that Confucian education and authoritarianism is uniquely necessary to govern the PRC because Chinese people are socialized to be conniving and selfish.
Interesting, because that hasn’t been my experience at all.
The Chinese people I’ve worked with and talked to (quite a few) almost uniformly see a practical, societal benefit to “Confucian” principles and are generally bemused as to why individualism fetishizing Americans often have a difficult time seeing their value.
“Conniving and selfish” actually strikes me as a very American-centric viewpoint of “Chinese people.” It seems necessarily rooted in an individualistic paradigm all the way down.
I don’t know what you mean by “American-centric viewpoint” but yeah all Chinese people in China and most ethnic Chinese abroad are taught about Confucian values and there’s not really much anyone would disagree with there besides maybe the patriarchy of it. The question is; how much do Chinese people believe other Chinese people embody those values. You’re supposed to respect your role in the ruler-subject relationship, but there were 7 or 8 rebellions in the Qing dynasty alone. The current government is 80-90 years old, younger than some Chinese people I know, and was a revolt against a government that itself began as a revolt less than 40 years before that. So yeah, how much respect is there for the ruler-subject relationship there? If the Chinese population were all that Confucian, why is the government still so afraid of revolt?
I have a few hundred in counterfeit bills in my wallet right now
> If the Chinese population were all that Confucian, why is the government still so afraid of revolt?
I think you’re wrongly reducing Confucian influence to only one dimension.
There are other countries with strong Confucian influence who have entirely different forms of government than China. Confucian influence doesn’t necessarily translate into sheeple.
> “You’re supposed to respect your role in the ruler-subject relationship…
When I’ve discussed Confucian influence with Chinese, the discussion has rarely even touched on “ruler-subject” dynamics. There are many more elements of Confucian influence – most prominently perhaps the relationship of the individual to the community without any particular reference to a “ruler-subject” framing.
The analects very explicitly enumerates only five fundamental relationships. Ruler-subject is one of them. Individual-society is not. No, it is not the entirety of the ideology.
My point is simply to question the extent to which Chinese society embodies Confucian ideals, the same way one might question “how free is America really?”
I can assure you, I am at least as familiar with Confucianism and Chinese culture as you are.
>“how free is America really?”
Free to get stormtroopered by the riot police when the Billionaires in the group chat stuck their hand up the NYC mayor puppet
https://wapo.st/3WSF7e2