“Predicting consumers’ choices in the age of the internet, AI, and almost perfect tracking: Some things change, the key challenges do not”

David Gal writes:

I wanted to share the attached paper on choice prediction that I recently co-authored with Itamar Simonson in case it’s of interest.

I think it’s somewhat related to your work on the Piranha Problem, in that it seems, in most cases, most of the explainable variation in people’s choices is accounted for by a few strong, stable tendencies (and these are often captured by variables that are relatively easy to identify).

I also wrote a brief commentary based on the article in Fortune.

And here’s the abstract to the paper:

Recent technology advances (e.g., tracking and “AI”) have led to claims and concerns regarding the ability of marketers to anticipate and predict consumer preferences with great accuracy. Here, we consider the predictive capabilities of both traditional techniques (e.g., conjoint analysis) and more recent tools (e.g., advanced machine learning methods) for predicting consumer choices. Our main conclusion is that for most of the more interesting consumer decisions, those that are “new” and non-habitual, prediction remains hard. In fact, in many cases, prediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante. Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise—so much so that it is often a waste of effort. We suggest marketers focus less on trying to predict consumer choices with great accuracy and more on how the information environment affects the choice of their products. We also discuss implcations for consumers and policymakers.

Sophisticated statistics is often a waste of effort . . . Oh no, that’s not a message that I want spread around. So please, everyone, keep quiet about this paper. Thanks!

10 thoughts on ““Predicting consumers’ choices in the age of the internet, AI, and almost perfect tracking: Some things change, the key challenges do not”

  1. This is an interesting paper and one I will use in future courses. However, I would call it a “polemic” on big data and machine learning. I think the evidence presented is important and the basic message about the difficulties of predicting consumer choices, and lack of improvement from recent machine learning methods, is refreshing. But I can’t shake the feeling that the evidence that is cited is selective. The prevalence of using these methods presents a question: are decision makers deluded, or are the successful uses of these more recent methods not generally reported? I do think there is a publication filter where the most successful uses in business are not likely to be published. So, I worry that the evidence the paper is citing is skewed towards examples where use of more data and/or more sophisticated methods is presenting only half of the story.

    As the old adage attributed to John Wanamaker says “Half the money I spend on advertising is wasted; the trouble is I don’t know which half”.

  2. I’m thinking back to previous posts on the limits of AI.

    I recall that interpretation of radiographs was discussed. I argued that radiographs are not feature-rich enough for AI, with significant issues being hidden in very slight differences in shading that a human eye can interpret but that are fundamentally difficult for AI. I named this the “Where’s Waldo Problem” because Where’s Waldo panels hide his face in complexity that can be instantly resolved by AI, the opposite of a very simple image that hides complexity like a radiograph.

    Another post described the limits of AI associated with designing micro-chips, specifically “floor planning” for chip layout. The limits here are different. Floor planning is extremely complex because of crosstalk/spacing and other interactions. In this case, AI seems limited in its ability to do “meta thinking” because it is trying to keep track of everything instead of making executive decisions about what to ignore. There are a lot of parallels to chess, where it should be pointed out that while computers can beat the best humans, they must consider orders of magnitude more moves before making a good decision. I call this the “Executive Reasoning Problem.”

    The issues with predicting consumer choice seem to be different from both of these inherent limitations. Here, AI seems to be unable to consistently tease a signal out of the noise because the nature of both the signal and the noise are constantly changing, being strongly influenced by unpredictable phenomena such as fashion and the information environment. Perhaps I should name this the “Greased Pig Problem.”

    I’m curious if others see things differently.

      • What are many journals? This is definitely not what i ever witnessed or read for journals in the social sciences. But what do I know, I stil enjoyed the article, but reporting such short timeframes signals wrong or inaccurate incentives…

  3. > In fact, in many cases, prediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante.

    This is exactly “tracking.” Large players can respond to these signals in near real time. Whether or not they have the opportunity to react (e.g., place an ad) depends on the purchase velocity.

  4. This is not a top journal, and it seems like this is review paper (and I guess the journal is mainly reviews), which was presumably invited, as these are prominent people in the area, and perhaps even editors had seen a prior working paper. So I don’t think this timeline is necessarily inaccurately short (but I could be wrong). More generally though, this is certainly not the timeline at the standard top journals in the social sciences, including in this field, though there are exceptions. To be clear, these are credible authors and I’m not saying anything against this paper, which I haven’t yet read.

  5. Seemed interesting. I have no experience here. In the abstract they say:

    > In fact, in many cases, prediction has become harder due to the increasing influence of just-in-time information (user reviews, online recommendations, new options, etc.) at the point of decision that can neither be measured nor anticipated ex ante. Sophisticated methods and “big data” can in certain contexts improve predictions, but usually only slightly, and prediction remains very imprecise—so much so that it is often a waste of effort.

    My read of the paper makes me think this line from later is more representative of their message though:

    > Moreover, the outsize influence of information near the point of decision in shaping consumer choices suggests that rather than attempting to measure stable consumer preferences and match their products accordingly, marketers must increasingly focus on how the information environment affects the choice of their products.

    The first blob sounds kind of defeatist, but I think the message of the second is to try to understand the just-in-time information better cuz the idea of fixed consumer preferences seems a bit flawed.

Leave a Reply

Your email address will not be published. Required fields are marked *