Bert Gunter writes:
This article in today’s NYTimes is a hoot, and might be grist for the lighter side of your blog … or maybe the heavier if you want to get into statistical fake detection, which is a big deal these days I guess.
The news article in question is called, “Five Stars, Zero Clue: Fighting the ‘Scourge’ of Fake Online Reviews,” subtitled, “Third parties pay writers for posts praising or panning hotels, restaurants and other places they never visited. How review sites like Yelp and Tripadvisor are trying to stop the flood.”
I agree with Gunter; it’s a fun article. Here’s my question: why aren’t there more fake reviews? Sure, lots of people are honest and would not cheat, but what I don’t understand is why the cheaters don’t cheat more. For example, suppose some crappy restaurant goes to the trouble of posting two fake five-star reviews. Why not go whole hog and post 100? Or would that be too easy to detect?
Or maybe we’ve reached an equilibrium. Right now if you’re looking for a place to eat, you can look at the reviews on google/yelp/tripadvisor/etc, and . . . they don’t give you zero information, but they provide a very weak signal. Not necessarily from cheating, just that tastes differ. But cheating muddies the waters enough, it just adds one more reason you can’t use these for much. So maybe the answer to the question, Why don’t they cheat more?, is that not much is to be gained by it.
Didn’t read the article due to the paywall (I read reviews at times, since they are “free”). But if your statement that businesses pay for fake reviews, then presumably they would pay more for more fake reviews. If so, then the marginal benefit of additional reviews probably declines relative to the marginal cost so there aren’t “more” fake reviews. In addition, to be at all effective, the fake reviews would probably need to not be identical – which increases the cost of making them, at least for many smaller businesses (I’m sure it is possible to automate the process and have the reviews differ enough to look independent, but that requires more sophistication than many smaller businesses are likely to have).
A possible related phenomenon is the number of 5 star reviews that are given either with no comments (which should be easy to provide more of) or the ones with negative comments. I particularly like the comments that say product X is the worst ever, but then give 5 stars. It reminds me of those course evaluations where students give the highest ratings for every question, but then have nothing to say when it comes to open-ended comments. I tend to completely discount those (as well as the lowest ratings without any comments), not knowing whether they checked off the right column or not. I have to be more careful if the evaluations are better designed so that some questions have “1” meaning a positive evaluation while other questions have “5” representing a negative one.
Any interaction with any reasonably large entity, n >1, will ask for your review. Sometimes, I am tempted to write a positive review because I am overwhelmed that an individual, for example, at the clinic, the internet provider, or the airline was helpful. However, should a subsequent encounter with the clinic, the internet provider or the airline go amiss, then I am certain any previous praise, stored somewhere in the bowels of some database, will be used against me when it comes to a dispute. Consequently, inaction is the wise go-to response to any request for ratings.
Perhaps (1) there are many more fake reviews, and we don’t recognize that they’re fake, or (2) it’s hard to make a fake review that doesn’t sound obviously fake. And of course (3), that a refreshingly large number of people don’t cheat, despite incentives to do so.
#2 will likely change due to generative AI tools. The next step will be fake reviews that sound real *and* that don’t all sound like each other.
#3 suggests that we ignore reviews authored by people named Ariely or Gino.
I guess one question might be whether at least in some contexts the fake reviews level out l, so there’s no real signal above the background signal of legit reviews.
When I’m review surfing, I try to look at the comments that suppoer the reviews. Not only would they be less likely to be faked (given the energy involved although as mentioned m, AI might level that out), but I think you can get info anyway that is helpful for evaluating products and their suitability for what I want irrespective of the number of stars assigned.
I also look for responses from proprietors to negative reviews. Often negative reviews tell you more about the reviewer than the “reviewee” (as I often self-sealingly felt about student reviews), and with respect to that possibility you get more context. And responding to reviews can suggest a sense of accountability from the proprietor..
I do that as well. Some of my favorites are the owners response to 5 star reviews that did not contain any verbal comments that say “thank you for your 5 star review.” That is almost as meaningless as the polls that ask you when you unsubscribe from something and list the reason “I no longer with to receive these messages.”
“I no longer wish to receive these messages” implies that at one time you did want to receive these messages. There’s usually another option like “I never signed up for these.” So _in principle_ correct answers to this question could be helpful. In practice I doubt anyone even looks at the stats.
Phil
I agree, but want to point out what I see as a common issue with almost all surveys. “I no longer wish to…” is clearly different from “I don’t wish to…” on close inspection. But I don’t believe that translates to a meaningful difference is survey responses. Most survey respondents do not read such questions closely – indeed, when I do answer those unsubscribe questions, I don’t even notice that “no longer” seems to imply that I once did – even if there is another choice that “I never signed up for….” Too many surveys mistake carefully designed questions for carefully designed responses. I believe many survey responses are actually expressing something different than what is being asked.
According to Schwab, Yelp’s earnings have grown at the stellar rate of -22.4% annually over the last five years. That probably explains the lack of fake reviews: reviews aren’t worth much to begin with. Yelp’s market cap is about $2.6B and it’s trading at 70x, so it’s TTM earnings are a measly $37M – thats for a world-wide advertising market for reviews.
Its earnings are equal to about five average home depot stores (2322 stores, $17.1B earnings, about $7M per store). It wouldn’t surprise me if the highest earning Home Depot store earns more than Yelp.
It would be interesting to know how much Yelp earns per review. I can’t imagine they want to try to hard to eliminate fake reviews, since their advertising dollar flow is directly dependent on engagement which is generated by reviews. Their onely incentive is to eliminate reviews that look fake enough to damage engagement.
FWIW: In 2022 Yelp had 20.9M reviews and $36.3M earnings, so they made about $1.75 / review after removing fakes.
Are ratings something that one should pay attention to? To me there are ratings by individual experts. When Nabokov says that Hemingway is a writer of books for boys, you should pay attention. Likewise, when Robert Parker says the Haut Brion 1945 is a 100, I’m willing to believe it. However, a lot of ratings, like Yelp, are done by amateurs. They tend to crowd at the asymptote. Book ratings on Amazon always have lots of four and fives. There are books deserving a five, but surely there are no more than three per year. I don’t want to bother rating every service or store that I patronize. Doctor ratings are completely ridiculous; the schmaltziest phonies do well. (I consistently got 4.7 which reflects the Amazon book rating effect.)
I give this posting a 3.8
A big problem with that concept is that someone who watches 300 new movies a year is not an expert on whether a movie will entertain someone who watches 3 new movies a year. A surgeon might have a pretty good idea whether another surgeon at the same hospital is good at surgery, but not beside manner or dealing with Ruritanian patients.
Chipmunk, Oncodoc:
Yeah, both your comments make me think that this post is a few years out of date. We used to look up restaurant reviews on tripadvisor or yelp or whatever when traveling, but I got burned enough times, sometimes by what must have been fraudulent reviews, but other times just by masses of low-quality reviews, that we’ve given up on this sort of crowdsourced review. So maybe one reason there aren’t more fake reviews is that nobody trusts them anymore. I still use Amazon reviews of products, not because I take the reviews so seriously but because I need some way to distinguish between otherwise indistinguishable choices.
Perhaps I am not typical, but I suspect I am. I still read restaurant and hotel reviews – frequently. And they influence me, despite all that I know about them being unreliable. But what is the alternative? How do I have any idea whether a new restaurant (one that I haven’t been to) is any good or not? I may find the quality of the review next to worthless, but what is the alternative? If I have a personal recommendation, that would certainly count for more, but that is often not available. I think the idea that the post is outdated and nobody uses these reviews is mostly wrong.
In fact, I’d go further. I read these product and service reviews even more than I used to. I like to think I’ve acquired some ability to distinguish between meaningless reviews and informative ones by paying close attention to the writing – the number of stars is not what I pay (much) attention to – but my increased experience using reviews makes me use them more, not less. Also, I’d have to say that I’ve been burned a number of times from low quality reviews, but more frequently they have helped me.
I suspect my experience is more typical than yours. The only thing that makes me wonder about that is the increased popularity of restaurant chains. I rarely visit them – if I did so more frequently, then I’d probably ignore the reviews since the food is relatively uniform from one brand X restaurant to another. But they appear to be immensely popular, so perhaps my experience is not like other people’s.
When I was in grad school, my then-girlfriend and I had very little money — the last several days of each month, before payday, we would often eat pasta with canned tomato sauce (Hunt’s was not bad, as canned tomato sauce goes) with some broccoli chopped up in it, or baked potatoes with melted cheddar cheese inside along with some spinach. When the paychecks came we would sometimes celebrate by going out to eat…at TGI Friday’s, or Olive Garden, or some other big chain. We had a good local pizza place, so if we went out for pizza we would usually go there, but for other restaurants the chains were a better value proposition. Bottomless soda or iced tea, plentiful portions of pretty tasty food, giant desserts of the type my wife calls “child’s desserts,” a type I still like (big and sweet and gooey).
After grad school I moved to the SF Bay Area, where we are blessed with thousands of good, local restaurants serving all genres of food. I haven’t been to a TGI Friday’s or an Olive Garden in probably 25 years. I certainly qualify as a ‘foodie’ now, willing to go extra distance or spend extra dollars to experience more variety and/or better food… plus I’m a sworn enemy of ‘factory farming’ of animals and would never eat the meat at those chain places. But I don’t mock the big chains or the people who eat at them. Been there, done that. A lot of places in the country, it’s hard to beat the value proposition.
This is reminding me of https://www.buzzfeednews.com/article/juliareinstein/marilyn-hagerty-anthony-bourdain-olive-garden
At any rate: like you, I don’t go to the big chain restaurants (and I agree there would be no point in checking the reviews), but for me it’s easy to understand the popularity.
I use reviews indirectly on Google Maps for restaurants all the time. I have a dietary restriction, so when I search for restaurants, it pulls up restaurants where either the menu mentions the option, or comments do. I have to actually read the comments though, since “has lots of gluten-free options” and “has no gluten-free options” are unhelpfully both surfaced when searching for gluten-free restaurants. So for my usage, the content of the review or other content (photos of menus) are what I’m using more directly. The fake reviews inflating the star level are of secondary impact.
I wonder if the fake reviews are limited in number just because they really only matter when there aren’t a lot of reviews. I wonder if restaurants even think of it as cheating or just seeding the reviews. No one wants to be that restaurant with a single review, and it’s a bad one.
I can think of one or two researchers who have made their name studying dishonesty and fraud … wonder what they’ve been up to recently…
I don’t use yelp, but for interpreting online reviews (amazon, etc) I developed what may be a novel strategy.
I wanted to get a new set of “bone conducting” headphones the other day. My current ones had a low max volume. So what I looked for was *negative* reviews from people complaining they were too loud.
Eg, if I was looking for a restaurant and saw complaints about “undercooked” tuna that would indicate they had what I wanted. Since I like it very lightly seared.
Anon:
That’s like the student at the University of California many years ago who decided to study with me because the professors in his applied statistics class told him not to. Don’t work with that guy, he’s too Bayesian!
Exactly. Positive reviews are standard, so that negative review carried far more weight than the positive version would have.
There is *a lot* of research in computer science about fake reviews of various kinds. I’d presume that this line of research traces all the way back to the early 2000s or so.
Today, the situation seems so dire that various companies and organizations have invested in techniques to spot fake reviews. Among these is Mozilla Foundation:
https://blog.mozilla.org/en/mozilla/fakespot-joins-mozilla-firefox-shopping-announcement/
A recent report on fake reviews: https://www.gov.uk/government/publications/investigating-the-prevalence-and-impact-of-fake-reviews. According to the report 11-15% of reviews are currently fake. In the spirit of this post, I guess the question is (as commonly said on Marginal Revolution): find the equilibrium. After you publish the first article on what the equilibrium looks like, the next article is how AI changes the equilibrium. Third article: fake peer reviews. This would be a great research area for those dishonesty experts.
Dale, this is a classic:
“Limitations
This study estimates the prevalence and impact of fake review text on consumer products alone. Therefore, these findings do not extend to services purchased online nor do they account for the impact of inflated star ratings. As a result, the true consumer detriment caused by fake reviews is likely to be higher.”
Paraphrasing: “Our methods are flawless. This presents the only risk to the accuracy of our findings is that we were much more lenient than we could have been, so while there is no liklihood that things are better than we claim, there is a major possibility that things are way worse than this study makes them out to be”
So here’s one of the key takeaways:
“well-written fake reviews distort consumer decision making – consumers were 3.1% more likely to purchase a product with well-written fake reviews”
‘Well-written’? No risk to discerning what’s ‘well-written’ from what’s not? Much less a ‘well-written’ fake review from a ‘well-written’ real revue?? OMG, that’s funny.
‘more likely’ than what? The same product at a different website? A different product at the same website? An A/B type study in which one arm shows the ‘well-written’ fake reviews and one does not? Than one without purportedly ‘well-written’ fake reviews, or a different product altogether?
What about the possibility that ‘well-written’ fake reviews use real reviews as their sources, but just write better? What about the possiblity that better products attract more ‘well-written’ fake reviews?
I don’t necessarily doubt the conclusions but the claim that there are effectively risks to the conclusions being accurate is hilarious, and is itself a good reason to doubt the quality of the work if not toss it out altogether.
sorry a little sloppy with the cut and paste but you get the gist…
You are correct to point out my lack of care in reading carefully. But then you did not really read the report, either. For example, they define well-written as either “subtle” or “strong” as follows:
“Subtle fake review text: Deliberate inclusion of one of the following elements that might raise
suspicion: overly vague or generic language, exaggerated language, several reviews left on
the same date or the same reviewer leaving two reviews. The following is an example of a
subtle fake review:
“Beautiful, sturdy, and meaningful! MAJOR transformation and definitely worth the
money. [reviewer name is “ProductReviewer”]
Strong fake review text: Deliberate inclusion of at least two of the following elements:
excessive capitalisation or punctuation, repetitive phrases and formatting, reviews covering
completely different products than the product listed on the page, acknowledgement of
financial compensation for positive reviews. In addition, the reviews were written by the open-
source text generation model GPT-2 (to mimic fake reviews written by bots). The following is
an example of a strong fake review:
“A must have machine for your everyday use… very easy to use and great quality
output… I was hesitant but it’s simple and easy and value for money!!! HOW do I GET
MY £20 VOUCHEER?”
Further, the study provides a great deal of detail and analysis – plenty to criticize, but at least enough to suggest that your wholesale mockery of the study is unwarranted.
“You are correct to point out my lack of care in reading carefully. ”
Apologies that wasn’t my intent. I didn’t have the idea that you had endorsed the study in any way; only that you offered it as an example of such a study. I don’t even know why I clicked on it but when I saw that description of risk I thought it was comical. I wasn’t criticising you, but I thought you might be interested in the surprising degree of nonchalance regarding the potential pitfals of the study.
“But then you did not really read the report, either. ”
Nope, I didn’t – nor claim to. As Andrew often says, detailing the bullshit takes so much longer than creating it that it’s usually not worth the trouble.