An odds ratio of 30, which they (sensibly) don’t believe

Florian Wickelmaier and Katharina Naumann write:

In a lab course, we came across a study on the influence of “hemispheric activation” on the framing effect in decision making by Todd McElroy and John J. Seta [Brain and Cognition 55 (2004) 572-580, doi:10.1016/j.bandc.2004.04.002]:

Two experiments were conducted to determine whether the functional specializations of the left and the right hemispheres would produce different responses to a traditional framing task. In Experiment 1, a behavioral task of finger tapping was used to induce asymmetrical activation of the respective hemispheres. In Experiment 2, a monaural listening procedure was used. In both experiments, the predicted results were found. Framing effects were found when the right hemisphere was selectively activated whereas they were not observed when the left hemisphere was selectively activated.

Two aspects of this study reminded us of recurring topics in your blog. [No, it was not cats, John Updike, or Jamaican beef patties; sorry! — AG]

Use of buzzwords: Why call it “hemispheric activation” when what participants did in Exp. 1 was tapping with their left versus right hand? This is a bit like saying “upper-body strength” instead of fat arms.

Unrealistic effect size: A 30-fold increased framing effect when tapping with your left hand (“right hemisphere activated”) Sounds like a lot. Even the foreign-language researchers claimed only a 2-fold increase (https://statmodeling.stat.columbia.edu/2015/09/22/i-do-not agree-with-the-view-that-being-convinced-an-effect-is-real-relieves-a-researcher-from-statistically-testing-it/). Maybe it was a combination of low power and selection by significance that rendered so large an effect?

Here are the original data (Tab. 1):

right-hand tapping left-hand tapping
safe risky safe risky
gain 8 4 12 1
loss 7 4 3 9

With right-hand tapping, the odds ratio is 8/4/(7/4) = 1.1 (no framing effect). With left-hand tapping, it is 12/1/(3/9) = 36. So the ratio of odds ratios is about 30.

We asked our students to try to replicate the experiment. We used an Edlin factor of about 0.1 for sample size calculation. Our data are 52/31/(26/57) = 3.7 with right-hand tapping and 56/27/(30/53) = 3.7 with left-hand tapping. The effect has vanished in the larger sample.

We think this makes a useful teaching example as it illustrates the now well-known limitations of a small-scale study with flashy results. We also see some progress because students increasingly become aware of these limitations and get the chance to learn how to avoid them in the future.

This reminds me of the 50 shades of gray study.

I agree that it’s good for students to be able to do these replication experiments themselves. Also good that we can start with a default skepticism about such claims, rather than having to first find some major problem in the study. Attention given to pizzagate-like irregularities should not distract us from the larger problem of hardworking scientists using bad research methods and getting bad conclusions. Remember, honesty and transparency are not enuf.

9 thoughts on “An odds ratio of 30, which they (sensibly) don’t believe

  1. My most recent experiments indicate that the presence of Jimmy Cliff on a movie screen increases the probability of eating Jamaican beef patties by 1.

    Here’s the DAG:

    Life —> Watching The Harder They Come —> ¡Patties!

    *The Author would like to express Jah Love to Natraliart and Matouk’s West Indian Hot Sauce.

  2. Are you suggesting that an odd ratio of 30 is suspicious by itself?

    Consider a low prevalence disease (p=2% in the general population,
    but p=11% among people with severe obesity ~15% of the population).
    You’ll get an odd ratio about 30.

    With a sample large enough(> 1370, 80% power), finding an odd ratio of 30 would be perfectly reasonable, no?

    • “With a sample large enough(> 1370, 80% power), finding an odd ratio of 30 would be perfectly reasonable, no?”

      For your example, I agree. But for a framing effect and with only n = 48, I think an odds ratio of 30 is unrealistic. What is really implausible, is the huge ratio of odds ratios “caused” by simply switching the hand.

Leave a Reply to Florian Wickelmaier Cancel reply

Your email address will not be published. Required fields are marked *