In answer to James Coyne’s question, no, I can’t make sense of this diagram.

Psychological Science  24(7) 1123-32, Fig. 2 copy

We last encountered James Coyne in the context of skepticism about claims that talk therapy can halve the rate of cancer recurrence, and skepticism about claims that giving plants to old people can extend life (of the people, that is, not the plants).

Now Coyne points us to a paper, “How positive emotions build physical health: Perceived positive social connections account for the upward spiral between positive emotions and vagal tone,” by Kok, B. E., Coffey, K. A., Cohn, M. A., Catalino, L. I., Vacharkulksemsuk, T., Algoe, S. B., . . . Fredrickson, B. L., from . . . Psychological Science.

Uh oh.

Coyne writes:

It took quite a while to figure out that the paper reported on a randomized trial with null results. And our re-analyses further show it was nonsense. Obviously, Fredrickson and colleagues don’t agree . . .

But the main issue is the complicated conceptual “figure” which she suggests shows how meditation will improve social relationships and health, setting off an upward spiral, like a flock of butterflies. [See image at top of post.]

Can you make sense of this diagram?

In some ways it looks like it’s supposed to be a mediational/moderator analysis, but I don’t think it corresponds to actual measurement points and I think some of the arrows are either misplaced or superfluous. The simple question that I’m asking you is does it really inform interpretation of the paper or is it just the flapping a butterfly wings to distract?

Here’s Coyne’s discussion (with James Heathers, Nicholas Brown, and Harris Friedman), and here’s the reply by Bethany Kok and Barbara Fredrickson.

12 thoughts on “In answer to James Coyne’s question, no, I can’t make sense of this diagram.

  1. It is a textbook structural equation model diagram (but for the black dot, afaik. Interaction?) Whether the latent variables have a real world meaning, and whether the model is used to obfuscate the lack of any interesting effect is a different matter. But the graph per se should not to be blamed for that.

      • Yes, textbook. Have a look for instance at Kline (2011): Principles and Practice of structural equation modeling. It explains the notation (sans the black dot) on p. 95. The black dot is most likely an interaction, if one interprets it as another node.

        The “actual” variables (in SEM literature they are called observed variables) are represented by the rectangles. The diagram does not express timing of the variables. It expresses (among other things) the conditional independence relations, such that if an arrow is missing between two variables X and Y, then they are independent given the set of all other variables. The big advantage of the graph is that it allows you to determine the minimum set of variables necessary for the indep. relation to hold – these are the direct parents of X and Y (ie all nodes with an arrow to X or Y).

        • The more I look at that black dot, the more strange I think it is, but I believe it is supposed to indicate moderation (based on the abstract). I don’t know why it needs a big black dot though. Maybe because it is both moderating and mediating?

  2. Haha, Mike, are you a con artist promoting loving kindness meditation or someone who got conned? This Kok snd Fredrickson paper doesn’t mention it is an RCT until the supplemental materials. And when we looked closely, we discovered it was a null trial p > .3.

    Do you believe the vagal tone data? We got a look st it and found some improbable outliers and any group differences were due to unexplained deterioration in the control condition. If you check Google Scholar, you can see thst Bethany Kok, the first author continues to publish on this topic now that she has left Fredrickson’s lab but no longer cites this paper. Lapse of memory, perhaps?

    • I actually have no idea what loving-kindness meditation is.

      To be fair, it doesn’t seem outrageous to me that taking steps to relax and have a more positive outlook on life benefits your health — but I absolutely agree with you that doing a convoluted analysis of a small sample with questionable measurements doesn’t do much to support that.

      One thing that stood out to me in their ‘rebuttal’ of your comments was the notion that while respiration period went up in the group of meditators, the difference wasn’t statistically significant, so it couldn’t explain the significant difference in vagal tone. Perhaps the garden-of-forking paths (as Andrew calls it) is bigger than we thought — even if a direct effect is not significant, you can always try to find a series of significant (p.05 links) that indirectly supports the hypothesis. Then you provide some justification for each link, and you have a paper.

      When you say you found ‘improbable outliers’, are you implying that the data may have been fabricated (to some degree)? Because then this discussion takes a completely different turn.

      • I’m one of the authors of Heathers et al., along with Jim Coyne. There is no suggestion that the data have been fabricated. If they had, there would presumably be at least some significant primary effects!

        Lead author James Heathers has written more about the problems with the Kok et al. paper here: https://medium.com/@jamesheathers (see his latest couple of posts). He explains in some detail how you need to be very careful in collecting and interpreting HRV data, because if you don’t, you end up with a mess, especially if you can’t recognise biologically implausible (or, in some cases, impossible) data and remove those participants.

  3. The path diagram is ridiculously over-complex for a small study of this kind, but I have greater concerns about the two bar charts that appear later on in the Kok et al. article. They occupy 1.5 pages between them, in an article in which there was apparently no room for a table of means and SDs of the principal variables; yet there is no explanation of what they really mean. About their only contribution is that, by counting the segments, one can learn that of the 65 participants who starred the study, only 52 were still proving data at the end; that’s another minor detail that was not deemed worthy of space anywhere in the text of the article.

Leave a Reply to Nick Brown Cancel reply

Your email address will not be published. Required fields are marked *