Imagining p<.05 triggers increased publication

We’ve all had that experience of going purposefully from one hypothesis to another, only to get there and forget why we made the journey. Four years ago, researcher Daryl Bem and his colleagues stripped this effect down, showing that the simple act of obtaining a statistically significant comparison induces publication in a top journal. Now statisticians at Columbia University, USA, have taken things further, demonstrating that merely imagining a statistically significant p-value is enough to trigger increased publication. . . .

The new study shows that this event division effect can occur in our imagination and doesn’t require literally seeing a pattern that reflects the general population. . . .

OK, I guess at this point you’ll want to see the original, a news article called “Imagining walking through a doorway triggers increased forgetting,” by Christian Jarrett in the British Psychological Society Research Digest:

We’ve all had that experience of going purposefully from one room to another, only to get there and forget why we made the journey. Four years ago, researcher Gabriel Radvansky and his colleagues stripped this effect down, showing that the simple act of passing through a doorway induces forgetting. Now psychologists at Knox College, USA, have taken things further, demonstrating that merely imagining walking through a doorway is enough to trigger increased forgetfulness. . . .

The new study shows that this event division effect can occur in our imagination and doesn’t require literally seeing a doorway and passing through it. . . .

Yes, I do find this funny. But, at the same time, I recognize that these are not easy questions. And, in particular, Jarrett is in a difficult position in that to some extent his job involves the promotion of psychology research, not just the evaluation.

I sometimes have a similar problem when blogging for the Monkey Cage political science blog. A bit of criticism of political science research is OK, but too much and I get pushback.

So, back to the research in question, Lawrence, Z., & Peterson, D. (2014). Mentally walking through doorways causes forgetting: The location updating effect and imagination Memory, 1-9 DOI: 10.1080/09658211.2014.980429.

Based on Jarrett’s description, I see a lot of red flags:

1. Lack of face validity. “Mentally walking through doorways causes forgetting”?? According to Jarrett, “The group who’d imagined passing through a doorway performed worse at the task than the first group who didn’t have to go through a doorway.” This could be true—all things are possible—but it sounds a little weird. And the researchers themselves seem to agree with me on this; see next point.

2. Claims that the effect is both expected and surprising. On one hand, “This effect of an imagined spatial boundary on forgetting is consistent with a related line of research that’s shown forgetting increases after temporal or other boundaries are described in narrative text.” On the other hand, the researchers write, “That walking through a doorway elicits forgetting is surprising because it is such a subtle perceptual feature . . . that simply imagining such a walk yields a similar result is even more surprising . . .”

This is what, following up on some observations from Jeremy Freese, we’ve called the scientific surprise two-step.

3. Lots of different small-n studies but no preregistered replications that I see. Lawrence and Peterson’s finding follows up on a paper by a couple of other researchers, four years ago, which, according to Jarrett, “shows that the simple act of walking through a doorway creates a new memory episode.” The recent paper and that earlier paper have a bunch of studies and comparisons, but it seems like a bit of a ramble (or what Freese calls “Columbian Inquiry”). Each time something interesting shows up, the researchers follow up with a new study that is evaluated in its own way with various idiosyncratic data-analysis choices.

Put it all together, and all I can say is: I’m not convinced. I’m not saying I’m sure these claims are wrong; I just think they’re pretty much at the same status as Nosek, Spies, Motyl’s “50 shades of gray” findings:

Participants from the political left, right and center (N = 1,979) completed a perceptual judgment task in which words were presented in different shades of gray. Participants had to click along a gradient representing grays from near black to near white to select a shade that matched the shade of the word. We calculated accuracy: How close to the actual shade did participants get? The results were stunning. Moderates perceived the shades of gray more accurately than extremists on the left and right (p = .01)

That is, before Nosek et al. tried their own preregistered replication:

We could not justify skipping replication on the grounds of feasibility or resource constraints. . . . We conducted a direct replication while we prepared the manuscript. We ran 1,300 participants, giving us .995 power to detect an effect of the original effect size at alpha = .05.

And got this:

The effect vanished (p = .59).

P.S. Just to be clear, I’m not trying to pick on Christian Jarrett. It’s not his job to evaluate the strength of claims that have been published in refereed psychology journals. We all just need to be aware that you can’t believe everything you see in the papers.

29 thoughts on “Imagining p<.05 triggers increased publication

  1. Weirdly enough, while I have a problem with the imaginary doorway I am willling to say that the real doorway and forgetting makes some kind of sense if only in my own anecdotal experience.
    I used to work in a kitchen with the serving lines through a doorway at one end of the kitchen and the dry goods storage through another door at the other end.

    Workers, myself included, were for ever standing in the kitchen or dry goods store say, “I’ve completely forgotten what I came here for”.

    Of course, I could suggest many other explanations but the doorway one is probably as good as any without more research.

    • Its hard to point out that something lacks experimental evidence without being taken as suggesting the claim is false and or not well motivated.

      If something is true, with adequate experimental inquiry it would replicate, there really is no alternative, no degree of well motivated-ness is sufficient if the claim is in any doubt.

      Some excerpts from Peirce “the First Stage of a scientific inquiry, resulting in a hypothesis of the very highest Plausibility,” “irresistible though it be to first intention, yet needs Probation [experimental evidence]”

      • Keith:

        You write, “It’s hard to point out that something lacks experimental evidence without being taken as suggesting the claim is false and or not well motivated.”

        If the claim is that “psychologists have demonstrated that merely imagining walking through a doorway is enough to trigger increased forgetfulness,” then, yes, I’m suggesting that claim is false.

        If the claim is that “merely imagining walking through a doorway is enough to trigger increased forgetfulness,” then I’d rather just say that this could be possible but I don’t see strong evidence for it.

        • Andrew:

          Communication of scientific criticism is challenging, to say the least.

          > “psychologists have demonstrated that ..”
          Of course, No!

          > “merely imagining walking through a doorway..”
          Yes, “could be possible but I don’t [no one should] see strong evidence for it.”

          But some of the comments were predictable – variations of _this make sense to me_

          But what (I think) you raised were only issues about the evidence for it.
          (Here I worry that folks are trying playing someone else’s game – which is OK if at some point you address/question the insight of the _winners_ of those games.)

    • Well, I often journey from my living room to my kitchen, and arrive unable to remember what I was seeking. Problem is, my home has an open layout on the ground floor and there are no doorways to pass through.

      Just sayin’.

      • @Clyde

        Good points. Zbyciclyts and Rahul make similar points below.

        If memory decays with time, then the door likely has nothing to do with it. Simply a delay between focus at time t and recall at time t+1.

        I’m just waiting for a psychologist to write the paper: “Walking through 1001 doors results in total amnesia”.

        I call such research _fiction science_ (FS). Unlike Science Fiction, which is fiction based on imagined scientific advances, Fiction Science is science based on fictions: Theoretical fiction (that doors cause forgetfulness) and methodological fictions (that the experimental design is anywhere near an adequate test of the theoretical fiction).

        As a branch of fiction writing I am looking for an English department willing to sponsor The International Journal of Fiction Science. We could also have courses on creative story telling, scientific props, and persuasive writing.

        • PS Just to be clear, I agree with Andrew the research could be true. However, my reason for classifying it as Fiction Science boils down to this:

          If your theory is incredible (e.g. close to fiction) then you need an incredibly credible experiment (close to fiction) in order to learn about the substantive hypothesis rather than the (in)adequacy of the research design.

          Thus, so long as psychologists continue to test incredible hypotheses — and unfortunately this is what current career incentives encourage — using at best somewhat credible research designs — and for some reasons this is often the case — then we must conclude that they are engaging in Fiction Science. Whether this is conscious or not I am not privy to.

        • Has it become out of fashion for academic psychologists to work on PTSD, schizophrenia, psychosis etc.?

          Publications seem to focus on fantastic results of a more trivial nature & questionable real world impact. The goal seems to make a reader go “Who would have thought! I don’t believe this!”

        • P(banal study|psychology study) is quite low I imagine. Psych is a huge field.

          P(psych study|banal study) is probably higher.

        • “If memory decays with time, then the door likely has nothing to do with it. Simply a delay between focus at time t and recall at time t+1.”

          With regards to experiment 1, they write:
          “The mean length of time to complete the mental walk was longer in the shift (M = 94 sec, SD = 25.4 sec) than the no shift condition (M = 70.5 sec, SD = 20.1 sec)…The inclusion of mental walk time as a covariate rendered the experimental condition a trending, though non-significant predictor (p = .17). Mental walk time itself was similarly a non-significant, trending predictor (p = .07).”

          Those mental walk times are interestingly long. The description sounds like the participants were told to walk directly from point A to point B: “They were told to imagine themselves in a particular part of the environment… and that they were to mentally walk to a different part of the environment.”

          They say the room was 30 x 60 ft so the max distance to the goal would be ~67 ft. The mental walk speed in the no-door group is then ~0.95 ft/s, compared to ~0.71 ft/s for the door group. Wikipedia gives preferred walking speed as ~1.4 m/s (~4.5 ft/s) corresponding to a walk time of ~15 seconds. So mental walk speed is ~5-6x slower than real walk speed?

          They do say “many participants in the shift condition focused a considerable amount of attention on the drapery separating the two rooms.” This makes it sound like they were mentally inspecting the drapes for ~25 seconds, which seems excessive. This would still would not explain the slow speed of the control group though.

          Additionally, this the most blatant example of “disproving strawman hypothesis A to accept substantive hypothesis B” I have seen to date:
          “Chi-square analysis revealed that participants in the shift condition were poorer at identifying the to-be-remembered image compared to the no shift condition, χ2 (1, N = 51) = 5.83, p = .016, Cramér’s V = .02. In other words, simply imagining walking through doorways resulted in more forgetting, consistent with the Event Horizon Model.”

        • I see now there are multiple papers on this. I was referring to this one:
          Zachary Lawrence & Daniel Peterson (2014): Mentally walking through doorways causes
          forgetting: The location updating effect and imagination, Memory, DOI: 10.1080/09658211.2014.980429

  2. Did you link to this report before? I swear I’ve seen it before, maybe behind a curtain in another room.

    My reaction was:

    1. How do they know people actually thought of what they were told to think of as they picked out the image? I always wonder this about imagining and thinking experiments: how do you know I’m thinking of the card in front of me?
    2. If the participants actually did try to follow directions, one group was told to remember something harder with more steps. That they phrase this as memories segmenting into rooms is less obvious as an explanation than that they added junk to the memory task. So my first reaction would have been that “more steps/harder memory” may affect other short-term memory. And if that was my reaction, I’d say “duh” because short-term memory has issues holding stuff and bringing it up for recall use.

    • @jonathan

      there is some subtlety here. You argue that the door is simply adding complexity to the task. Above I have argued that it ads time. In both cases, however, the door still causes fading of memories. It is just that it is (fully?) mediated by complexity or time.

      There are two interesting aspects here:

      1. Form what I gather the original paper has a different mediation story, whereby door leads to compartmentalization, leads to forget.

      2. Even if we grant them that, I find the conclusion sensationalist. It is not quite the same thing to say: “compartmentalization causes forget” than “crossing a door causes forget”. The forced nature of this is seen form the quote Andrew makes above about “subtle perceptual feature”. Diving a room with a wall and adding a door is – where compartmentalization is concerned – hardly a subtle perceptual feature. Just ask any interior designer.

  3. It drives me nuts that Radvansky framed the original research as being about “forgetting”. The narrative-comprehension literature (in psychology) shows repeatedly that when there is some kind of “episodic shift” in a story, information associated with the pre-shift episode is somewhat less well-remembered than the same information is if there is no such shift. This is a sensible finding in that when we’re reading (or otherwise experiencing a narrative), information from the current episode is more available in memory than information from a prior episode; if this weren’t the case, working memory would soon become overloaded and our ability to build a mental representation of a narrative that has information integrated into appropriate units would be compromised. That this kind of process occurs during real life (as opposed to in our understanding of narratives) is unsurprising. Our memory is not an undifferentiated mass of information. It’s organized in a variety of ways, and our access to this information ebbs and flows in some predictable ways. To the extent that going from one room to another is associated with a change in “episodes” (e.g., in that room I was watching TV; in this room I’m making guacamole), it’s a boring fact of the way the mind works that prior-episode information is not quite as easy to access as current-episode information.

  4. So what is the effect size? How much worse did one group perform than the other? Isn’t that crucial?

    Why do science writers imagine that “Group-1 performed worse than Group-2” is sufficient detail? Would, say, a comparative review of laptops or cars be as non quantitative?

  5. If I were to generalize on the basis of doing no research myself (and the opportunity cost of reading much of this literature is too high) I would say that any unrelated task interferes with memory. You are distracted, you forget. If you are more distracted, you forget more.

    Check figure 1 in the paper, which shows experiment 1. This isn’t a normal doorway, it’s an opening made by drapery. How can you call this a “doorway” when clearly there’s no way you can hang a door on it?

    Experiment 2 uses virtual rooms in computer software. Another highly natural environment. (figure 4) Unless we’re characters in a Sims game, we aren’t walking through this type of doorway.

    So go back to my first paragraph. The weirder the situation, the more distraction. The more distraction, the more forgetting. That distraction leads to more forgetting seems reasonably intuitive and rather parsimonious. Maybe not publishable, though.

    Full text is available free here: http://www.tandfonline.com/doi/abs/10.1080/09658211.2014.980429#.VRWYWOGvM6E

      • Rahul
        I could not get your link to work so this might be the same thing:
        http://www.theinvisiblegorilla.com/videos.html
        This is a famous example and your comment made me wonder whether the paper could be discovering another form of the same thing (and like zbycyclist I don’t have the time or inclination to read the paper carefully). Any experiment where subjects know they are being watched is likely to make them concentrate on something – and that concentration can easily make them forget or not see other things. I’m not sure the study really is showing what it claims to be showing.

      • Related tasks interfere with perception. The gorilla video shows the blocking effect of concentration. The famous “Paris in the the spring” triangle where you skip the repetition. A common internet image of a page spelled weirdly that asks “fi yuo cna raed tihs,” and so on.

  6. Even if so, your explanation (& zbicyclist’s too I suppose) says there is some effect whatever the explanation.

    I’m more cynical. I think they are reading stuff in noise. I’m saying this cannot even be replicated. Change, the lab, time, cohort & this effect will vanish.

  7. There is a genuine psychological effect of context well established in memory. You recall better if asked to do so in the same context as the one in which you learned. The real door effect seems to be re-discovering the context effect and therefore is of no additional information to the literature other than it’s a cute description of a context effect with a fanciful causal mechanism proposed.

    Since all this really just happens in our brains what matters fundamentally is the perceived context. If you have a vivid imagination and generate a new context then you will most likely generate a weaker form of the effect as well.

    Of course, I should probably read the papers to check but I’m willing to bet quite a bit that a pre-registered effect can be found and it depends on how vivid your imagination is. I also bet it’s pretty small and relatively uninteresting.

  8. > Now psychologists at Knox College, USA, have taken things further, demonstrating that merely imagining walking through a doorway is enough to trigger increased forgetfulness…

    Well, we all know that people who hunker down in their office all day are the ones who get things done and the ones who venture out and engage the world (i.e., ones who go through doorways) lose all higher cortical function within a couple months. Actually, there’s a study for you – lock a bunch of people in solitary for a couple years and see if they’re less forgetful than the control group afterwards. (What is the control group for that experiment?) Mmm… but perhaps keeping them in solitary confinement is problematic. What other doorway-free environments are there? Ah, I know, keep them outdoors for a couple years and see if they turn out less forgetful.

    /facepalm/ Honestly, what constructive purpose does an experiment to determine whether “imagining walking through a doorway triggers increased forgetting” serve? Suppose it does? What non-pointless things would one do with that information? Jeezus bleeping christ, have we run out of important things to investigate?

    Richard Hamming, “You and Your Research” – http://www.cs.virginia.edu/~robins/YouAndYourResearch.html

  9. I don’t read much of the psychology literature and I am, in general, incredibly skeptical of most of the findings. However, the little exposure I’ve had suggests that a large percentage of the literature is pure ad-hoc speculation, justified by previous ad-hoc speculations.

  10. Here’s a cool experiment I did today after reading this post (I was in bed all day, down with a viral infection, nothing to do but just lie there). I wanted to do a simple experiment with an easy-to-measure result that involves imagining stuff, to see what effect it has on me.

    I was able to increase my pulse by 5 beats per minute (from 63) by just thinking about running really hard for about 30 seconds. I could not raise my blood pressure though, which was a surprise (it actually fell, although it’s probably just measurement error). I didn’t try to replicate it since I am fundamentally a psycholinguist by training.

    I think that another cool experiment might be to just imagine raising your arms in a victory pose, the way Amy Cuddy recommends, and measure the pico-gram increase in testosterone. I would definitely do this experiment if I had the measurement instruments (I don’t know how to measure testosterone).

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *