Learning from self-experimentation

Seth Roberts is a professor of psychology at Berkeley who has used self-experimentation to generate and study hypotheses about sleep, mood, and nutrition. He wrote an article in Behavioral and Brain Sciences describing ten of his self-experiments. Some of his findings:

Seeing faces in the morning on television decreased mood in the evening and improved mood the next day . . . Standing 8 hours per day reduced early awakening and made sleep more restorative . . . Drinking unflavored fructose water caused a large weight loss that has lasted more than 1 year . . .

As Seth describes it, self-experimentation generates new hypotheses and is also an inexpensive way to test and modify them. One of the commenters, Sigrid Glenn, points out that this is particularly true with long-term series of measurements that it might be difficult to do on experimental volunteers.

Heated discussion

Behavioral and Brain Sciences is a journal of discussion papers, and this one had 13 commmenters and a response by Roberts. About half the commenters love the paper and half hate it. My favorite “hate it” comment is by David Booth, who writes, “Roberts can swap anecdotes with his readers for a very long time, but scientific understanding is not advanced until a literature-informed hypothesis is tested between or within groups in a fully controlled design shown to be double-blind.” Tough talk, and controlled experiments are great (recall the example of the effects of estrogen therapy), but Booth is being far too restrictive. Useful hypotheses are not always “literature-informed,” and lots has been learned scientifically by experiments without controls and blindness. This “NIH” model of science is fine but certainly is not all-encompassing (a point made in Cabanac’s discussion of the Roberts paper).

The negative commenters were mostly upset by the lack of controls and blinding in self-experiments, whereas the positive commenters focused on individual variation, and the possibility of self-monitoring to establish effective treatments (for example, for smoking cessation) for individuals.

In his response, Roberts discusses the various ways in which self-experimentation fits into the landscape of scientific methods.

My comments

I liked the paper. I followed the usual strategy with discussion papers and read the commentary and the response first. This was all interesting, but then when I went back to read the paper I was really impressed, first by all the data (over 50 (that’s right, 50) scatterplots of different data he had gathered), and second by the discussion and interpretation of his findings in the context of the literature in psychology, biology, and medicine.

The article has as much information as is in many books, and it could easily be expanded into a book (“Self-experimentation as a Way of Life”?). Anyway, reading the article and discussions led me to a few thoughts which maybe Seth or someone else could answer.

First, Seth’s 10 experiments were pretty cool. But they took ten years to do. It seems that little happened for the first five years or so, but then there were some big successes. It would be helpful to know if he started doing something in last five years that made his methods more effective. If someone else wants to start self-experimenting, is there a way to skip over those five slow years?

Second, his results on depression and weight control, if they turn out to generalize to many others, are huge. What’s the next step? Might there be a justification for relatively large controlled studies (for example, on 100 or 200 volunteers, randomly assigned to different treatments)? Even if the treatments are not yet perfected, I’d think that a successful controlled trial would be a big convincer which could lead to greater happiness for many people.

Third, as some of the commenters pointed out, good self-experimentation includes manipulations (that is, experimentation) but also careful and dense measurements–“self-surveillance”. If I were to start self-experimentation, I might start with self-surveillance, partly because the results of passive measurements might themselves suggest ideas. All of us do some self-experimentation now and then (trying different diets, exercise regimens, work strategies, and soon). Where I suspect that we fall short is in the discipline of regular measurements for a long enough period of time.

Finally, what does this all say about how we should do science? How can self-experimentation and related semi-formal methods of scientific inquiry be integrated into the larger scientific enterprise? What is the point where researchers should jump to a larger controlled trial? Seth talks about the benefits of proceeding slowly and learning in detail, but if you have an idea that something might really work, there are benefits in learning more about it sooner.

P.S. Some of Seth’s follow-up studies on volunteers are described here (for some reason, this document is not linked to from Seth’s webpage, but it’s referred to in his Behavioral and Brain Sciences article).

11 thoughts on “Learning from self-experimentation

  1. Interesting post. Thanks for the references! I'd consider the article successful if only because it raises such basic questions.

  2. Thank you for your comments, Andrew. To answer your questions:

    “Is there a way to skip over those five slow years [at the beginning where not much happened]?” I don’t think the first five years were less productive than the rest. During those years I found connections between breakfast and sleep, processed food and weight, and faces and mood, not to mention realizing the value of self-experimentation.

    “What’s the next step [in the study of depression and weight, given my results]?” I believe the next step, when doing research, should be as small as possible. In this case, it’s to do n = 1 studies with persons other than myself.

    “Might there be a justification for relatively large controlled studies (for example, on 100 or 200 volunteers, randomly assigned to different treatments)? Even if the treatments are not yet perfected, I'd think that a successful controlled trial would be a big convincer which could lead to greater happiness for many people.” The larger the study, the more it differs from what you’ve already done, the more untested assumptions being made. The more untested assumptions being made, the greater the chance that at least one of them is wrong. If one of them is wrong, the whole endeavour fails – becomes a big waste of time and money. Which are limited.

    “What does all this say about how we should do science?” I think it says that we should pay more attention to idea generation – how to generate ideas worth testing. Compared to idea testing, idea generation is undervalued and underemphasized. Exploratory data analysis is about generating ideas during data analysis, but the chance that a piece of research will generate ideas also depends on what is done before data analysis – on experimental design, for example.

    “All of us do some self-experimentation now and then (trying different diets, exercise regimens, work strategies, and so on). Where I suspect that we fall short is in the discipline of regular measurements for a long enough period of time.” Very good question: where do “we [Andrew et al.] fall short”? I think that lack of regular measurements is part of the answer. I also think the answer has two more parts, which interact. First, “chance favors the prepared mind.” I knew a lot more than the average person about depression, even though it wasn’t my specialty within psychology. And I knew a lot more than the average person about weight control, even though that wasn’t my specialty area, either. The main reason is that I had taught introductory psychology and covered both topics. Second, I was (and am) an outsider in the areas of depression and weight control, with nothing to lose by proposing “crazy” ideas. (Thorstein Veblen wrote an essay that argued that Jewish scientists were so successful because they were outsiders.) Yet I am enough of an insider to know how to publish something and to hope to gain professionally by doing so. So it’s a balancing act: close enough to have a rich knowledge base, close enough to know how to publish and to benefit by publication, far enough away to publish something highly unorthodox.

  3. Hi Seth. Your answer to my last question is interesting because it suggests that people other than doctors or psychologists might have difficulty doing effective self-experimentation. So now I don't feel so bad about not doing it! I guess to really understand the phenomenon, I should do some self-experimentation myself. I'm just too lazy, I think. A structure is needed. That's what's good about the blog–it "forces" me to put down some thoughts every day.

    OK, now for my real question. It seems to me that there's a cost-benefit issue. Your treatments for depression and obesity are pretty simple–the obeseity treatment is really simple. And it has seemed to work for you and others. So why not do the big study?

    You write, "The larger the study, the more it differs from what you’ve already done, the more untested assumptions being made." Why must this be true? Can't you just use the exact treatments that you've developed so far?

    On the minus side, doing a big expt all at once increases cost and reduces flexibility and opportunities for learning. On the plus side, it could be a powerful "convincer" that could help people right away. Given that your treatments are pretty well defined right now, and given that it's gone slowly working with one person at a time, I'd guess that the pluses of the big study outweigh the minuses at this point.

  4. “So why not do the big study?” Because it is likely to be a big waste of time and money. But that’s just restating what I’ve already said. You go on to write:

    “You write, "The larger the study, the more it differs from what you’ve already done, the more untested assumptions being made." Why must this be true? Can't you just use the exact treatments that you've developed so far?”

    No, I can’t. Because it is unclear what “the exact treatments” are. Consider the vastly simpler case of simply studying a person other than myself. Suppose I get up at 6 am. I start watching faces on TV at 7 am. Person X, my second subject (subject number one was me), gets up at 7 am. Should X start watching faces at 7 am or 8 am? See how non-obvious it gets right away? Because no two people are alike? Should X watch exactly what I watched? Probably not, because X has different interests. I haven’t even mentioned recruitment, training, and data collection, all of which are vastly more complicated with more subjects. Consider recruitment. Any recruitment scheme makes many assumptions. If any are wrong, the whole thing may fail – recruitment will be too slow. I can think of no reason that I am likely to guess correctly repeatedly about this stuff. Without experience, assumptions are guesses.

    Do you know of any examples where something jumped successfully from very small (n = 1) to very large (n = 100), skipping intervening steps?

  5. Hmm . . . I don't know the answer to your last question. I certainly read about a lot of n=100ish studies without much of a sense of what came before. My guess is that they are preceded by small-n studies that are much less formal than your self-experimentation. Actually, you've already tried your depression and weight-loss treatments on a few volunteers, so you're not quite starting at n=1 anymore.

    Perhaps Dave Krantz or my sister Susan could comment on how things are done in cognitive psychology. My impression is that there is a lot of playing around with ideas before the large study, but that the large study still involves a leap into the unknown.

    But back to the details of your studies. What about the weight-loss treatment? That seems pretty straightforward–drink X amount of sugar water once a day, separated by at least an hour from any meals. To do a formal study, you'd have to think a bit about what would be a good control treatment (and then there are some statistical-power issues, for example in deciding whether it's worth trying to estimate a dose-response relation for X), but the treatment itself seems well defined.

    And, yes, recruitment won't be easy, and other complications will certainly arise. But, on the plus side, you could learn faster by studying many people at once. And then if the treatment is demonstrated to be effective, it could help more people sooner. I'd think that these two potential benefits would outweigh the costs of doing the study and the risks of coming up with a dud.

  6. Question for Seth Roberts: So what exactly IS the diet? How much sugar water, and/or olive oil, etc., per day, and when and how often, please? How much weight loss should be expected? And is the sugar water or olive oil continued after the weight is lost? Thank you.

  7. Sir,I am interested to know exactly how much water with how many tablespoons of frusctose did you consume each day and for how long? Did you have any signs of diabetes? Or any other symptoms?

    Carter

  8. For those who advocate a trial of 100 or more participants, both you and they didn't mention who might pay for it. Some group of experienced researchers has to organize, plan, and conduct the trial, then massage the resulting raw data to produce a meaningful report to disseminate to some undefined audience. Who would benefit enough to pay the costs of a trial the result of which would be what — yes, it worked for me or no, it didn't? Ughhhhhh!

  9. Bob,

    Seth had a similar comment to me once: he thought that academic scientists worried too much about getting federal funding, and he felt that, in many ways, research could be improved if it was not funded, since then the researcher would have to think harder about each data collection effort.

Comments are closed.