Skip to content
 

Another update on the spam email study

I think youall are probably getting sick of this by now so I’ll put it all below the fold.

Akinola Modupe and Katherine Milkman responded to my email about their study:

We want to clarify the reason we believe that the use of deception and a lack of informed consent were appropriate and ethical for this research study. In this project, we were studying how the timing of a decision affects discrimination based on race and/or gender. The emails all participants in our study received were identical except for a) the sender’s name (we used 20 names that pretesting revealed were strongly associated with being either Caucasian, Black, Indian, Chinese or Hispanic, as well as associated with being male or female) and b) whether the meeting requested was for today or for a week from today. Recipients were randomly selected and were randomly assigned to one of the race/gender/timing conditions. This study design will allow us to test for baseline levels of discrimination in academia (e.g., are faculty more responsive to Lamar Washington than to Brad Anderson?) as well as whether bias is more extreme when people make decisions that affect them immediately or in the future (e.g., does Lamar Washington fare better against Brad Anderson when he makes a request for the future than for today?). No individual response is meaningful (nor could individual institutions be singled out due to small samples at each school studied), but the overall patterns of responses will allow us to test for potential bias.

To study discrimination, it is crucial that study participants are not aware they are being observed, as this knowledge has been shown to dramatically change peoples’ behavior. Thus, we obtained permission from our IRBs to employ deception. To study discrimination, it is also crucial that informed consent be waived. If participants were offered an opportunity to opt into our research study even without knowing its purpose, it would likely have biased our sample towards including only those professors who would agree to a meeting. “Audit” research employing deception and a waiver of informed consent has been conducted many times before, and in fact, typically with far more costs to participants. Past research of this type has examined discrimination in housing markets, employment markets, car purchases, and mortgage applications (see for example Ayres & Siegelman (1995), Bertrand & Mullainathan (2004), Kenney & Wissoker (1994), Neumark et al. (1996), Pager, Western & Bonikowski (2009), and Yinger (1986)).

There were two potential downsides to our study that some of the postings on your blog have highlighted and that we attempted to mitigate with our experimental design. In some cases, our study (1) absorbed faculty time and (2) created a potential negative externality (our study may have reduced faculty members’ likelihood of responding to future messages from prospective PhD students). For the former inconvenience (absorbed time), we apologize sincerely. We believe that the importance of gaining an understanding of discrimination in academia is worth the sacrifice of faculty time imposed by our request for a quickly-cancelled 10-minute meeting. Regarding the latter issue (a negative externality), we hope faculty will not neglect emails from prospective PhD students in the future as a result of our study. The extremely high response rate received to our meeting requests confirmed our belief that academics are among the most generous professionals, and we hope you and others will not reduce your generosity as a result of one extremely atypical email.

My final thoughts on the matter:

1. I have no problem with the IRB’s approving the study. I think it’s great that IRB’s are there to protect people receiving dangerous drugs or other potentially serious interventions, but I don’t think that they should be worried about little things like this. In general, I think the whole IRB thing is out of control and tend to oppose suggestions that IRB’s should be restricting non-invasive research.

2. I suppose one reason I was particularly sensitive is that, as a survey researcher, I already feel a bit uncomfortable about the way in which pollsters get people to respond for free.

3. As noted in my earlier blog entries, I have no problem whatsoever with the study’s use of deception. What I didn’t like–and still am not thrilled with–is that they got our participation without our permission, and then did not compensate us for our effort. As I noted in one of the comment threads, I can respect the view that responding to an email is a minor effort that should not be worth $10–but I’d like to be the judge of that! In general, I’d think you’d get happier participants if you asked permission ahead of time or compensated afterward. I remain uncomfortable with the idea of taking people’s labor for free in this way.

4. As also discussed, it would not be hard to ask for permission in a study that involves deception, for example by sending an email several weeks in advance saying that they are b-school professors doing a research project and that they will contact me in a few weeks by email. In the initial contact, they don’t give info on the topic of the study. Then, if I agree to participate, the rest of the study goes as planned, and I’d have no idea that the purported student’s email is fake. Or the researchers could do the study as is, and then offer some compensation at the end. Again, if the study is worth bothering 6300 people (without asking for their permission), I think it’s worth compensating these people a bit.

5. To return to point 1, I certainly don’t feel the people who did this study should be punished in any way, nor do I feel that the IRB’s were asleep at the switch. (I never suggested such a thing, nor do I think I’m any position to do so. I emphasize this only because, based on the comments to the earlier entries, some people seemed to think that was what I was suggesting.) I think it’s fine to approve this sort of study, and I also favor ethics guidelines under which it would be expected to ask the permission of research subjects, or to compensate them afterward (or both).

12 Comments

  1. William Ockham says:

    I think you are being entirely too easy on these researchers. What they did is theft and the fact that they had the best of intentions is no excuse. The end doesn't justify the means. They clearly think that what they stole from you is of trivial value, but it is the only irreplaceable commodity in existence. I'm sure that for many of the people involved the time stolen was minimal, but in a study this large, inevitably some of the "participants" will lose a significant opportunity.

    The mealy-mouthed reply you received is almost laughable. My kids use these excuses. Their reply boils down to "it was for a good cause" and "all the other kids were doing it". They want to hide behind jargon, but deception is lying and "lack of informed consent" is stealing. "Two potential downsides" were actually completely predictable outcomes. "Absorbed" is a nice way to avoid taking responsibility for the effects of their actions. And, of course, they have to use "potential negative externality" which is classic economist's jargon for the damage they've done.

    Here's the challenge I'd like to give the researchers. Are you willing to sacrifice 10 minutes of your life for every person you involved in you study? That's about 1050 hours. I'm sure there are good soup kitchens in Manhattan and Philadelphia that could use the help.

  2. JR says:

    I really am amazed at the reaction of you and the other commenter. You lost a tiny amount of time -far less than you've used moaning about it! (although it seems that you've enjoyed that, so at least it wasn't "wasted").

    Really, the authors points seem reasonably well thought out. Maybe they could have got consent some way in advance; but it hardly seems like a big deal that they didn't.

    And really, could you even 'see' $10 if it went into your bank account? I'd be willing to bet it'd get lost in the 'income' pile, and you'd never know it was there. the only thing the $10 idea would do is massively increase the cost on the researchers.

    And in general I've always been really uncomfortable with paying for survey responses. I worry that people will see it as a 'liberty enhancing coercive offer' (as Joel Feinberg might have called them). For poorer people (and especially those on the breadline), they just might not be able to afford to say 'no' to $10 or $20 for half an hour at night. And if you're paying them, they'll probably feel that they should respond to questions they feel uncomfortable with, because you're paying them. Far better to just ask for their time for free, giving them the maximal opportunity to control the situation.

  3. Andrew Gelman says:

    JR:

    1. Indeed, spending an hour at a task of my own choosing is far more pleasant than spending 5 minutes on something I did not choose to do.

    Regarding this latest blog entry: I thought it was only fair for me to post the researchers' email to present their side of the story on this blog too. I also did warn people not to read this latest entry if they were getting tired of the topic!

    2. Perhaps you're right about the $10; however I would prefer to be the judge of that, rather than others deciding how I should value my time. I feel similarly about survey respondents in general: it is hardly coercive, in my opinion, to offer money to someone to answer some questions.

  4. JR says:

    "it is hardly coercive, in my opinion, to offer money to someone to answer some questions"

    For me, where the coercion comes in is where people really need the money (even if only $10). Once you make the offer of $n in exchange for answering y questions, they have two options – accept or decline. I would argue that the 'decline' option isn't 'real', at least for those people who really need the money. If you need the money so badly that you'd consent to anything, I think you're being coerced.

    Now it might be that in a country which has really high decomodification this situation doesn't arise, but in the English speaking world (at least) it's not that hard to find people who genuinely struggle to buy food each week.

    Feinberg used the example of a woman who could not afford to pay for life-saving medical care for her child. A millionaire offered to pay for it on the condition that she married him. She had no realistic way to decline, and so was coerced, even though formally she was freer than before (because she gained another choice). The 'cash for questions' situation is far less severe, but it can still fundamentally undermine the dignity of respondents.

    We didn't create the situation, but I'm uncomfortable exploiting it (and in nationally representative surveys, I'd be willing to be that there are more than 0 people in this situation).

  5. Megan Pledger says:

    But a survey is different. If a person doesn't want to answer a question then they can just skip the question or answer "don't know" or lie.

    That's the real problem with offering money – you get poorer responses.

    Anyway, it's part of American culture to disregard small sums of money as meaningless. Other coutries have different cultural values about small sums of money especially earned money.

    This is a real study where deception should have been used.
    http://edition.cnn.com/2010/US/05/13/doll.study/i

    The kids are asked which cartoon picture describes an attribute when the only difference between pictures is skin colour.

    The skin colours are put on a continium of hue so there's inherent racial bias. The (young) kids spot the pattern and give the questioner the answer the kid's thinks the questioner wants. The face order should really have been randomised for each kid.

    The kids are asked to pick one ("which kid would you like to play with?"). The kids don't get the option of "all of them" or "none of them". (Although an older kids who sees through the survey gives that answer. He also looks slightly affronted that they are trying to dupe him into being racist.) The kids should have been asked something like "put in this circle all the kids you'd like to play with".

    The young kids are going to pick the picture that look like them if they have that attribute. If a young kid has been told "you're realy bright" recently and then asked to pick out the bright kid then they will pick out the kid who looks like them. S/He's not identifying the colour with a "race" but with her/himself.

    This study is so flawed. It gets a result that you'd kinda expect but you can't trust.

    (I'm not sure how a deception study on kids would get through an ethics committee though!)

  6. Sebastian says:

    JR – for good survey's they actually pay you _before_ you fill out the survey and regardless of if you do. I've seen this a number of times – I'm part of a longitudinal education study back home in Germany and they send along 20 Euros with the questionnaire last time.
    I believe Knowledge Network does the same.
    The point here is to make the respondent feel valued by the researchers – not explicitly to pay her/him for the time.

    Note that something similar would have been the case here – results wouldn't have been affected by the compensation, because it would have occurred independent (or at least after) the data was gathered.

    I'm still not sure how I feel about the whole thing, mainly because I do think that some of the earlier studies they cite are really, really important, but I'm quite sympathetic to Andrews concern – namely that we need to make sure it remains costly enough to bother people if we don't want to end up bothering people too much.

  7. Zubon says:

    If being paid to take a survey "fundamentally undermines dignity," I can only imagine how traumatic it must be to be "coerced" to work for 40+ years to pay for food and shelter.

  8. Matt says:

    I'm very glad to see that this experiment has been getting attention. I was one of its unknowing subjects , and also one of those who was very upset by it.

    I'm not upset because of a lack of compensation for lost time. I'm upset because it was a deliberately false e-mail that exploited the fact that professors will dedicate their time and attention to inquiries from prospective students. Faculty now have every reason to be suspicious of future e-mails by prospective students, which is a real harm to those students.

    I'm even more angry about the professors' response. The comment by "Ockham" is dead right: it's a childish, half-assed pseudo-apology that minimizes complaints by saying they came from a "small fraction" of participants. Thanks, guys, for trivializing my concerns!

    Did they get any NSF funding for this? I wonder if Congress might like to know how taxpayer dollars were spent.

  9. still annoyed says:

    When I wrote to K. Milkman to complain about having rearranged part of my day in order to meet with "Chang Huang", the fake student who essentially stood me up, I specifically asked her *not* to contact me again in the future, not even in response to my complaint message. This was because of how angry her debriefing message made me, and how hard it is for me to focus on important tasks like grading and dealing with ACTUAL students when I feel that way. It's crunch time and I do not need the distraction.

    Anyway, here's what I got today, in addition to a form letter from the two IRBs acknowledging their receipt of my complaint. Note that they received "many supportive and critical" messages. Luckily I'm not angry anymore– I guess laughable is the right word at this point.

    Dear Colleagues,

    We are writing to follow-up on your concerned response to our debriefing message regarding a study we recently conducted. We deeply apologize if this study negatively affected you. Please note that we have compiled and reflected upon the many supportive and critical responses we have received, and we have thoroughly briefed our respective IRBs and university officials about your messages. We felt it was important that we do this before responding to any individual email. Our IRBs are very aware of your concerns and would be happy to speak with you if you so desire. Attached is a joint message from the Columbia University and University of Pennsylvania IRBs designed to address your questions. If you would like to learn more about the work we are doing and our explanation for it, please click here: http://opim.wharton.upenn.edu/~kmilkman/Website_P

    Thank you for taking the time to share your perspective on this study.

    Sincerely,

    Katherine L. Milkman and Modupe Akinola

  10. William Ockham says:

    People who keep focusing on the small amount of time per participant really don't get it. Here's an analogy that explains how I feel about this. Suppose I want to do a survey to discover what coins men carry in their right front pants pocket. I could pay a pickpocket to steal change from guys in public places. Of course, I wouldn't keep any of the money (let's just say I donate it all to charity). This is really no big deal, right? Nobody's gonna starve because they are missing their pocket change? But what about the people who needed that change to take the bus home? What about the guy carrying the 1969-S Doubled Die Obverse Lincoln Cent (worth something like $85,000)? If he didn't know he was carrying it, did I steal a penny from him or $85,000?

    The central problem is that I can't know exactly what effect I'll have on people by stealing their pocket change, but if I steal from enough people, someone will suffer some serious harm. The same thing is true when you starting stealing time from people.

  11. anon says:

    One comment on your point 4 – any attempt to get consent before the study is sure to introduce bias. The profs who respond to a 'will you take part in my study' email are by definition those who are helpful, and are not necessarily a representative sample of all profs. So if a study which measures 'helpfulness' as an outcome only includes profs who you already know are more helpful than average, it can't give valid results. I'm afraid the only way to get consent for this kind of thing without messing up the study design is retrospective consent. And maybe retrospective payment too, I'm neutral on that one.

  12. Richard says:

    As one of the spammed professors, I was also outraged by the Milkman et al. study. Let me offer several points:
    1. the study was poorly organized. The fake email to me said that the (fake) student was very interested in my research. Yet although my department does have a PhD program, my specialty (as well as the specialties of another 50% of the the dept) is specifically excluded (with my happy consent) from the PhD track. So any student purporting to want to come and get a PhD with me is either stupid or spam. Upon receiving the email I promptly ignored it as evidence of an unaware grad student spam campaign. I was wrong of course, but only in its origin, not in its essential nature.
    2. The shifting rationale offered by Milkman et al. The initial debriefing email claimed that the point of the study was to study decision making, particularly decisions to act 'now' versus 'later'; while this message mentioned discrimination, my (and my colleagues') reading of it felt that it focused primarily on behavior and response, with the issue of 'under-represented' groups as a secondary focus. Yet after the 'participants' grew angry, a second email (quoted above) shifted the focus of the study's rationale to discrimination; decision-making is now the secondary focus of the study. This seems to me (and to many others) a rather weaselly way of attempting to post facto justify a very bad decision.

    I am not particularly concerned with compensation, nor even with the time I wasted on this issue (as I said, I dismissed the email as spam when it first arrived). I am extremely angry at the sense of distrust that this 'study' has already sparked in me and others. As the director of graduate study in a medium sized department, I get dozens of emails every day from prospective students. Milkman et al. have inflicted serious damage on both my willingness to answer those emails as assiduously as I have to date done, and on my ability to trust what the 'students' are asking me. For this alone she and her fellow researcher deserve my opprobrium, and the academy's censure.