The Other Side of the Night

Don Green points us to this quantitative/qualitative meta-analysis he did with Betsy Levy Paluck and Seth Green. The paper begins:

This paper evaluates the state of contact hypothesis research from a policy perspective. Building on Pettigrew and Tropp’s (2006) influential meta-analysis, we assemble all intergroup contact studies that feature random assignment and delayed outcome measures, of which there are 27 in total. . . . We find the evidence . . . consistent with Pettigrew and Tropp’s (2006) conclusion that contact “typically reduces prejudice.” At the same time, our meta-analysis suggests that contact’s effects vary, with interventions directed at ethnic or racial prejudice generating substantially weaker effects. Moreover, our inventory of relevant studies reveals important gaps, most notably the absence of studies addressing adults’ racial or ethnic prejudice, an important limitation for both theory and policy. We also call attention to the lack of research that systematically investigates the scope conditions suggested by Allport (1954) under which contact is most influential.

I like that they don’t just give their conclusions; they also talk about the limitations of their data.

The contact hypothesis is a big deal, and I know that Don Green and his collaborators have been thinking a lot about meta-analysis in recent years, so I’m glad to see this research being done.

I’ve not yet read the article, but I did notice that it doesn’t have a lot of graphs—and those that it does have, are close to impossible to read. This just seems to be a preprint, though, so maybe someone can help them visualize their data and their findings, and some real graphs can be added to the paper before it is published in its final version. With all this work on data gathering and data analysis, it would be a pity for the results to not be explored. There should be lots more stuff to be uncovered from some graphical displays.

P.S. The second author of the paper, Seth Green, is a Columbia political science graduate and is currently working at a startup called “Code Ocean.” Googling led to this webpage which says nothing about Columbia. Apparently Code Ocean is “a cloud-based computational reproducibility platform” with a mission to “to make the world’s scientific code more reusable, executable and reproducible.”

It also says on that webpage that Code Ocean handles code in Python, R, and eight other programming languages. If so, I think they should handle Stan code too! In all seriousness, I think a bit of Stan would help this project, in that it would push the researchers toward modeling the effects of interest, and away from time-wasters such as this: “We begin our quantitative analysis by assessing cross-study heterogeneity using Cochran’s Q. The test decisively rejects the null hypothesis of homogeneity of effects across studies (Q(26) = 173.563, p < .001; I^2 = 0.85). We therefore reject the fixed-effects meta-analysis model in favor of a random-effects meta-analysis model, where the variance of the normal random component is estimated using method of moments. The resulting estimate is 0.39, with a 95% confidence interval ranging from 0.234 to 0.555.” Or this: “The results presented in Table 3 suggest that a one-unit increase in standard error is associated with a 2.07 unit increase in effect size, although the pattern is of borderline significance (two-tailed p-value = .049).” I know they’re just doing their best, and I like the general flow of this analysis; it would just be easier in Stan to get to the substance.

Traditionally, classical analyses based on statistical significance have been considered to be the safe option when analyzing data in the social sciences. But as we continue to work in areas where data are sparse and noisy, and effects are highly variable, it will make sense to just start with Bayes, to say goodbye forever to “p less than .05,” and to use Stan as your first rather than last resort for managing uncertainty and variation.

P.P.S. Cat picture demonstrating contact hypothesis from Diana Senechal.

7 thoughts on “The Other Side of the Night

  1. Hello all,

    Seth Green the co-author author and not the red-headed actor here.

    1) Sorry the plots are tiny. They are saved as PDFs within the PDF so they should get big when you zoom in. If you check out the accompanying Code Ocean page, , you can see them in full-screen, pop-out-window glory.

    2) All feedback on the paper, the presentation, etc. is most welcome!

    Thanks Andy for your feedback and for the blog space.


    • Seth:

      If you do things right, your graphs should be readable on the page, no zooming required. These particular graphs are not so complicated; with careful design you can make them readable as is.

      Also, since we’re on the topic:
      – Figure 1 is labeled “Effect Sizes.” I think this should be “Estimated Effect Sizes.” Estimates are all you ever have.
      – Figure 2 is pretty unreadable, except for that very small (I assume) subset of your readers who know what is meant by “Boisjoy 06 B” and all the rest. I think you’d be much better off with a scatterplot with something relevant about these studies, not just these cryptic names. Also, again, the effect sizes are estimates.

      I’m not trying to be harsh here, just helpful. I recommend you add a coauthor to your paper whose entire job is visualization. I think this could add a lot to your findings. And, why not? Don Green is advising a bunch of poli sci students at Columbia: he could have a couple of them take my class this fall on Communicating Data and Statistics, and one of them could then help you on this paper.

      • Hi Andy, thank you for the comments. “Boisjoly B” is indeed unclear, we fixed that in the paper but not in the Code Ocean submission, thanks for catching that. “Estimated Effect Sizes” is indeed more accurate, we will fix that as well before publication. Also we are taking a step back and incorporating your feedback more generally and thinking about new ways to visualize and analyze these studies, this has all been very helpful.

        – Seth

    • In general, try to make the reader’s job as easy as possible — don’t make them do your work for you. Time is short and figures are the quickest way to get a sense of the paper’s message — if they’re not easy to read, many readers will just move on to the next paper. Few will take the time to zoom in, fewer will click on yet another web page…

      Even just increasing the font size in the figures and maybe changing the figure shape will go a long ways.

      It’s a pity and a lot of people do it — both in papers and in slide shows — they’ve done all this hard work to build a story and then they hide it by making it hard for others to see!

      • Hi anonymous, you are totally right, this is one of those true points that’s obvious in retrospect but escaped me as we were editing a million details on the way to getting the paper out..Thanks for reading!

  2. Argh – it ate the beginning – the full comment repeated below.

    “(Q(26) = 173.563, p _less than_ .001; I^2 = 0.85). We therefore reject the fixed-effects meta-analysis model in favor of a random-effects meta-analysis model”

    Yuck! at least the developers of I^2 warned against the “strategy (fixed or random effects method) is determined by the result of a statistical test” and I had the good sense to withdraw from this and or earlier papers on it ;-)

    Stan would help this project, in that it would push the researchers toward modeling the effects of interest
    Well you do need to consider if a more informed analysis would be helpful given how the data was generated and became available in publications.

    Unavoidably? having to do this “common analytic strategy for meta-analyses in the social sciences is to calculate standardized mean difference” takes you most if not all the way to p_values. Is commonality of size across relevant studies reasonable to assume or is it just commonality of direction of effect? Is there any reasonable way to get defensible likelihoods that might have partially pool-able parameters?

    It would be nice to have a Stan vignette for Don Rubin’s effect surface meta-analysis ideas and other likelihood based analyses – but I think mostly it would show that with data generated and made available in publications like here – there’s mostly just uncertainty.

Leave a Reply

Your email address will not be published.