Encouraged by the success of his self-experimentation to help his sleep, mood, and weight concerns, Seth Roberts has been experimenting with the effects of drinking flaxseed oil. Here’s an example of his results:
Commenting on another recent one of Seth’s self-experiments, I wrote,
Seth,
Not to be a wet blanket or anything, but aren’t you worried that your findings might be due to expectation effects: you knew which oil you were taking when doing the tests, right?
Seth replied,
Andrew, no, I’m not worried that the results are due to expectations. If the results always conformed to my expectations, I’d be worried, but they haven’t — see my post about eggs. Moreover, this particular result confirms a result that was a surprise. In other words, I’ve gotten the same result when I was expecting it and when I wasn’t expecting it.
I’m still concerned, though. Seth is saying that it’s not just an expectation effect because he wasn’t always expecting the results. But I could see a bias arising from positive feedback, as follows: You try a new treatment and then see what happens after, with no expectations except that things might change. There is some noise to this measurement–just at random, it will be higher or lower than before. Having seen this, you adjust your expectations; this then affects your next measurement, etc.
I’m not saying this is definitely happening, but it could be.
To Seth: maybe you could get a partner in experimentation, someone who lives or works nearby, and he or she could give you a randomly assigned oil. That is, your partner would know which oil you’re getting, but you wouldn’t. In fact, you wouldn’t even know if you were being given something new that day. (It wouldn’t be hard to set up some complicated randomization scheme so that, for example, you would get one oil for several days, then another, etc.) You, of course, could provide the same service for your self-experimentation partner. This also has the virtue that you’ll get twice as many measurements.
This would be more persuasive if there were evidence that expectations are powerful. Most of the evidence suggests they aren't.
I agree, I should do "blind" tests. However, because of that evidence, they are low on my to-do list. The harder question is what to do next.
Expectations aren't powerful? What about the placebo/nocebo effect?
http://www.ncbi.nlm.nih.gov/sites/entrez?cmd=Retr…
P-ter,
Thanks for the link. I remember a news article a couple years ago saying that there was no placebo effect–this would seem to have an important impact on how we teach biostatistics–but then I assume that the placebo effect has been studied before, so it's hard to believe it's really zero. Something worth looking into, I suppose. In any case, Seth's balance and reaction-time measurements are indeed "continuous subjective outcomes," so I don't think he's off the hook.
In any case, Seth's balance and reaction-time measurements are indeed "continuous subjective outcomes," so I don't think he's off the hook.
true, true. doing blind tests (or double blind–put the oils in identical containers, covering the labels, and have the partner pull one at random?) would make the experiments more rigorous.
I read the article summarizing the studies of placebo effect and showing none as well. It is interesting, but seems to stand in contract to the studies looking at alcohol consumption in a party setting. There the study was to look at the effect on behaviour and reported inebriation in subjects drinking punch either with or without alcohol. In people with little or no experience with drunkenness, alcohol was required for them to report a response. For people with prior experience with drunkenness no alcohol in the punch was required. It suggests that in situation in which a neural or psychological effect is being measured there is a possibility of a placebo effect, but perhaps not in other areas of biological study. Sorry, I do not have access to the reference, perhaps someone else remembers the paper. It was published somewhere reasonably prominent, Science comes to mind as a guess.
So if the placebo effect doesn't exist, what does that mean for homeopathy? ;)
'I’m not worried that the results are due to expectations. If the results always conformed to my expectations, I’d be worried, but they haven’t'
Seth's comment doesn't really make sense at all. Observing that some of the results did not conform to expectations doesn't say much: whether expectations play a major role or not has nothing to do with whether some results will 'have a different sign' to the a priori expected one. Even if an expectation acts by exerting a 'gravitational pool' to the observed result, it is perfectly reasonable that some outcomes will go the other way.
And an example in case my point is not clear: Assume you expect a certain measurement to give you a value of 5, and instead you get a value of -2. It is impossible to tell what effect a different expectation would have: perhaps if you expected -1 the result would be -2 again, or it would be -30. Or, indeed, 16 (why not?). Add a stochastic element to the DGP of the outcome, and there are bound to be some times when the result has a different sign to the expected one simply by chance.