Mark Evanier is a great storyteller. Here’s his story from 2011 about the unpromising topic of cold-call marketing surveys:
I [Evanier] don’t like salespersons or survey-takers who phone me. . . . I especially don’t want to answer questions from survey-takers because I figure they’re calling to build up a profile on me…and that profile will be used somehow to try and sell me things I don’t want.
Sometimes, I immediately ask the caller, “Is the last question about how much money this household makes?” Because they always want to know that and they usually hide it at the end. I’m not going to answer their questions either way but if that question’s in there, I’m especially not going to answer their questions.
The other day, a lady phoned and told me she was conducting a “brief survey” and would just need a few minutes of my time. Before I could ask her about the last question, she said, “For every survey that is completed, a donation will be made to the Susan G. Komen Breast Cancer Foundation”…which I believe is no longer even the name of that organization. So instantly I suspect they might not really be dealing in any way with the Foundation or maybe they’re keeping in the word “cancer” to ratchet up the sympathy. I asked the lady, “How much?” and from there on, it went pretty much like this. To her credit, she started giggling about halfway through . . .
You can click through to read the whole thing. The quick summary is: (a) it doesn’t sound like much of a donation is going on, and (b) the last question on the survey was indeed how much his household makes.
As Evanier says, this is not the fault of the survey interviewer (soon to be replaced by a robot, if that has not already happened). At the same time, it’s not Evanier’s fault that his time got wasted on this call. Ok, in this case not wasted because he got a good story out of it, which I’m now sharing with you. But, yeah, most of the time, it’s time wasted.
These things are scams on multiple levels. They con you into spending 45 minutes of your time giving them free data, they they con the companies that hire them into thinking they’re providing them with useful marketing intel. In this case there was the extra con of pretending they’d be meaningfully supporting a charity. The big problem is they want your survey responses but they don’t want to pay for it.
I wrote about this in one of our very first blog posts, back in 2004:
The U.S. is over-polled. You might have noticed this during the recent election campaign when national polls were performed roughly every 2 seconds. . . .
My complaint is not new, but this recent campaign was particularly irritating because it became commonplace for people to average batches of polls to get more accurate estimators. As news consumers, we’re like gluttons stuffing our faces with 5 potato chips at a time, just grabbing them out of the bag.
In recent years, as polling has proliferated, response rates have been going down. Why bother responding at all? Bob Groves and others have done research in this area. One reason to respond is to be helpful and civic-minded.
The recent proliferation of polls—whether for marketing or to just to sell newspapers—exploits people’s civic-mindedness. Polling and polling and polling until all the potential respondents get tired—it’s like draining the aquifer to grow alfalfa in the desert, or dredging all the crabs out of the bay—a short-sighted squandering of a resource that should be renewable.
I’d forgotten that poll averaging was already a thing back in 2004.
This all seems very naive to me. I do not participate in polls – ever. I have even stopped being polite to the caller (who is often a computer these days). Everything Andrew says I agree with – but it seems at odds with prior statements he has made about political polling. I find it hard to identify a middle ground where some polls are useful and others exploitative.
I’ll add one more reason to not engage will polls/surveys. Even under the best of circumstances (a legitimate organization with clear intentions to elicit my beliefs), it is so unlikely that the results will be interpreted correctly that I am providing a response that I then have no control over how it is used. Will the results be reported accurately, with appropriate caveats? How do I know?
Regarding polls and surveys, I just say no. Never. Not under any circumstances.
I’ve laid down a fairly extreme position – anybody care to carve out when and where it makes sense to respond to polls or surveys?
I also always wonder who is answering these polls. Answering such calls has been actively dangerous for awhile now.
Has the popularity of chatGPT/deepfakes made enough of an impression on people yet though? Giving away your voiceprint to a random caller is setting yourself up for so-called identity theft (the bank/etc gets defrauded but somehow this becomes your problem).
When you get a spam call, best practice is to not answer at all. Don’t even reject the call as that provides info that the line is actively monitored.
>I also always wonder who is answering these polls.
Me too. I don’t know anyone who answers an unknown number on their phone anymore. Not even the older folks I know. I would imagine that the people answering phone calls and then agreeing to surveys have some pretty key differences vs people who don’t…
I always answer my phone.
Dale:
Pollsters now sometimes use internet panels: they recruit participants, pay them, and then occasionally poll them. That seems cool to me.
The only time I’d recommend answering a cold-call poll is if it is for some public purpose, for example a government social, economic, or public health survey.
And how would you really know? Remember there are entire industries whose whole purpose is to call people and scam them. So they might easily just say “we’ve been commissioned by the FBI to find out your experience with online fraud” or whatever
That’s a good point. Usually, this type of surveys have a website with further information about the study for the selected participants, like this one from CPS: https://www.bls.gov/respondents/cps/
Typically, interviewers are instructed to provide this information, especially if respondents question about the legitimacy of the survey.
I suppose there are reasons to believe that responses from paid panels are more valid than unpaid cold calls. In fact, I do believe that. But there is still the problem that respondents have no real incentive to be truthful or to think carefully about their responses. It is virtually impossible for me to think of ways to induce respondents to carefully consider the questions and respond accurately. To some extent, government surveys still elicit such behavior, but since trust in government appears to be at an all time low, I’m not sure that will continue much longer.
‘Never’ seems extreme. Would you make an exception for official statistics surveys?
Dale,
I often use survey data for my research, so I would be a bit of a free rider if I categorically refused to participate in surveys. Of course, if someone cold called me on the phone, I’d probably wouldn’t participate.
I do regularly participate in a government sponsored panel survey. When I was selected, they sent information about the survey and a line to follow-up with questions, offered to send someone to my house so I could do the survey in person, or let me do it over the phone, or let me do it online (the option I’ve chosen). I’m paid 40 Euros every year for completing the survey. I suspect that if you were selected for this sort of survey, you’d also participate.
Josh and Peter
I wouldn’t participate for 40 Euros, but I do have some price at which I’d do it for the money. Yes, “never” is too extreme – some government surveys are not really voluntary and I’d participate if chosen (like jury duty). But that’s about it. More importantly than individual participation, however, is my lingering concern that there is really no way to provide binding incentives for people to be truthful and thoughtful in their responses. As with many things, we rely on trust for that – and our collective ability to earn trust seems to be eroding more quickly than our ability to design surveys.
I have a landline phone, whose number I enter whenever I have to give a phone number online, but which is always disconnected except when I order a pizza or have to make a service call. That’s my response to cold-callers, pollers or otherwise. Survey forms in the mail go right into the recycling bin.
There are probably some legitimate causes I am missing but prior to this strategy I started to fill out one too many push-polls.
I was a student in the early 2000s, when it was still common in Budapest for pollsters to knock on your door. I was studying for an exam, but the person claimed that “it will only take 10 minutes”. I was very naive, and I thought, why not, I could do with a break.
But of course it took about 45 minutes. The pollster insisted on reading all the questions and options, instead of giving me the sheets to do it myself. He was a slow reader. So… 45 minutes of my life I will never get back, but a valuable life lesson: a reasonable net gain! The funny thing is that it was about investments in securities, and at the time I was a hand-to-mouth consumer with no savings whatsoever, so I am not sure why they asked me.
These days I only answer polls
1. online,
2. where I can see all the questions from the very beginning,
3. I care about the organization who is making them. Eg recently I answered the poll where the city I live in is asking about where to spend money (more trees, more bike stands, stuff like this). I find it neat that I can participate in these decisions.
I also answer one-question polls, eg rating deliveries from 1 to 5.
In the EU with GDPR, it has become pretty difficult to use this data for covert profiling, so I think that has stopped.
About those one question ratings: can anybody explain these two odd things? First, many ratings (for hotels, golf courses, restaurants, etc.) are given 5 stars and the provider responds by thanking for the excellent review – but there is no review other than the stars. Second, I’ve seen many 5 star reviews that do contain content – but the content is all negative. The respondent had a horrible experience but then gives a 5 star rating (by the way, this second situation reminds me of student course evaluations where I was never sure whether the extreme ratings represented what the student actually thought – was 1 on a scale of 5 a great or a terrible rating?).
Andrew refers to this Mark Evanier story taking place in 2011; Susan G. Komen was peripherally mentioned , so I found this, and note the 2012 reference to the State (Georgia) very much in the news yesterday and today–as well as the perennial target
https://www.washingtonpost.com/national/health-science/timeline-of-key-events-in-komen-controversy/2012/02/07/gIQAX4EWxQ_story.html
“● January 2011 – After an unsuccessful run for Georgia governor, Karen Handel joins the Susan G. Komen Foundation as a consultant. She was then hired as a senior vice president for public policy. During the Georgia primary, Handel’s platform included advocating for Planned Parenthood’s defunding.”
Interesting to revisit this! I wonder if we are at the end or near the end of this type of polling proliferation, at least in the market research world, given all the other sources of information and way to gather those type of data.
It would be a win-win for everyone: 1) people in general that don’t have to be bother to answer many of these polls, 2) the surveys themselves that might benefit from having respondents not being burden by this over-polling phenomena, 3) and even the companies that can get these data for cheaper.
Good luck with that. If you’ve bought anything within the past 5 years (and I mean anything), you almost immediately get surveyed on your purchase. And if you don’t respond, then you get asked again and again until you figure out how to unsubscribe and then hope that you actually get unsubscribed. My impression is that the over-polling and over-surveying problem is the worst it has ever been. It is possible that this is the extreme and the pendulum will shift, but it does have a bit of a tragedy of the commons element. We might collectively gain from having less polls/surveys, but it isn’t in anybody’s interest to eliminate theirs unless they can be guaranteed that everyone else will as well.
Dale Lehman is absolutely correct in what he just wrote–except for “less polls/surveys” which should be “fewer polls/surveys” because that subtlety was grilled into us during our 1940s primary school days. One poll/survey we never had back then, and indeed did not exist until after I exited grad school, was the student evaluation of the instructor. I imagine for younger contributors to this blog, it might come as a shock to realize that this poll/survey does not go back to the days of the McGuffey Readers
https://en.wikipedia.org/wiki/McGuffey_Readers
“About 120 million copies of McGuffey’s Readers were sold between 1836 and 1960, placing its sales in a category with the Bible and Webster’s Dictionary. Since 1961, they have continued to sell at a rate of some 30,000 copies a year.”
That Wikipedia factoid regarding current sales can’t possibly be correct. In fact, when I clicked on it, this came up: A picture of a skunk and the following statement—“Well, that stinks. We can’t find that page.”
Survey non-response in discussed in part VI (“Leftovers”) of this post from Stuart Buck so decidedly on-topic for this blog that I’m surprised Andrew hasn’t already blogged it (or has he already written a post on a delay to publication?):
https://goodscience.substack.com/p/metascience-since-2012-a-personal