Gur Huberman writes:
The following questions crossed my mind as I read this news article.
1. Is there an observable corresponding to the term “public opinion?”
2 From the article,
Polls in Russia, or any other authoritarian country, are an imprecise measure of opinion because respondents will often tell pollsters what they think the government wants to hear. Pollsters often ask questions indirectly to try to elicit more honest responses, but they remain difficult to accurately gauge.
What is the information conveyed by the adjective “authoritarian?”
My reply: Yeah, there’s a Heisenberg uncertainty thing going on, where the act of measurement (here, asking the survey question) affects the response. In rough analogy to physical quantum uncertainty, you can try to ask the question in a very non-invasive way and then get a very noisy response, or you can ask the question more invasively, in which case you can get a stable response but it can be far from the respondent’s initial state.
To put it another way: yes, it can be hard to get an accurate survey response if respondents are afraid of saying the wrong thing; also, there’s not always an underlying opinion to be measured. It depends on the item: sometimes there’s a clearly defined true answer or well-defined opinion, sometimes not.
In this case, the news article reports on a company that “tries to address this shortcoming by constantly gathering data from small local internet forums, social media companies and messaging apps to determine public sentiment.” I’m not sure how “public sentiment” is defined here; they seem to be measuring something, but I’m not quite sure what it is.
This happens a lot in social science, and science in general. We have something of importance that we’d like to measure but is not clearly defined, we then measure something related to it, and then we have to think about what we’ve learned. Another example is legislative redistricting, where researchers have come up with various computational procedures to simulate randomly drawn districts, but without there being any underlying model for such a distribution. And I guess lots of examples in biomedicine where researchers work with some measure of general health, without the latent concept ever being defined.
I think the problem is comparable to one of the issues with NHST. When a test is performed and the null is rejected, the erroneous conclusion is reached that this means the favored alternative (whatever that happens to be) is supported. In the case of public sentiment, we want to know how people (all, some?) feel about X but there are reasons we can’t ask them directly and find out how they feel (for example, we want to know how they feel about a particular religion but suspect they won’t be truthful). So we ask them about Y – which is related to X. We have a hypothesis about how question Y might be related to question X – but, unfortunately, that relationship is never really tested. So, we ask about Y and then make conclusions about X. However, the chain of logic from Y to X is something that may be evident to the designer of the survey, but not necessarily to the respondent, particularly if the respondent isn’t really invested in thinking carefully about how they “really” feel. As a result, when we get answers to Y, I’m not sure we even get meaningful information about Y, and even less sure we find out anything about X. And, the more subtle the wording of the question Y (to attempt to avoid pitfalls about influencing people’s responses), the worse the problem is.
For example, suppose we ask how comfortable a person would be living next door to a family with religion R as an indirect way to see how they feel about religion R. They respond with a 2 out of 10 (where 1 = not at all comfortable). We then interpret that as a measure of their attitude to religion R. But there are several intermediate steps: “living next door” brings in issues associated with religious practices (e.g., loud prayer calls at 5AM), living arrangements (apartments, houses, etc.), feelings about living in close proximity to anybody of any religion, etc. So, I’m not really sure what their rating of 2 really says about their feelings about religion R. And, I’m even less sure that they’ve thought carefully about the wording of the question. I have similar concerns about the Implicit Bias Test, where the prompts reveal “something” about implicit bias, but it isn’t clear to me how this translates to actual behavior.
I guess I’d rather be a two-handed economist than a multi-handed pollster.
Quote from above: “I have similar concerns about the Implicit Bias Test, where the prompts reveal “something” about implicit bias, but it isn’t clear to me how this translates to actual behavior.”
They should design an Implicit Association Task concerning the Implicit Association Task. Perhaps people’s, including social scientists’, attitudes towards this all can subsequently be found to be associated with attitudes and behavior like:
“I think it’s perfectly fine for social scientists to spend decades performing research on the IAT without first finding out what it is they are exaclty investigating”
“I support members of the public having mandatory training-sessions, which might be heavily influenced by research using the IAT, concerning things like implicit associations at work even though I really don’t know the validty and/or long-term consequences of this all”
“I think it’s perfectly fine for social scientists to attempt to nudge or steer people and/or societies in a way that aligns with their own particular view on life and society”
Some more possible attitudes to possibly investigate with an IAT-IAT:
“It’s perfectly fine for social scientists to design and use a vague concept and ‘objective test’ to measure it if that concept can subsequently be used for something I value”
“I never really think about whether social scientists’ research on certain topics might actually highlight these certain topics and result in negative consequences”
“I never really think about the possibility that some things might better be improved by focusing on other things”
“Even if some social scientists have a God-complex, that doesn’t mean their research and writing should be seen in the light of this possibility”
“I, as a social scientist, won’t stop until every single instance and form of X is gone. If that means that more of X will be present in the future as a result of my research and writing and actions that is just the price I will have to pay”
This is a common misconception. You’re talking about an observer effect. The Heisenberg uncertainty principle isn’t an observer effect, it’s a fundamental property of waves (classical waves too). No matter how gentle your measurement is, there is a tradeoff between measuring, for example, the position and the speed of a wave. To measure the position of one peak in the wave, you have to ‘zoom in’ on it and so you lose information about the speed. To measure the speed, you need to focus on many periods of the wave, and so you don’t know the position. Again, you need not affect the wave in a relevant way to get this effect.
https://scienceexchange.caltech.edu/topics/quantum-science-explained/uncertainty-principle
https://chem.libretexts.org/Bookshelves/Physical_and_Theoretical_Chemistry_Textbook_Maps/Supplemental_Modules_(Physical_and_Theoretical_Chemistry)/Quantum_Mechanics/09._The_Hydrogen_Atom/Atomic_Theory/Electrons_in_Atoms/Uncertainty_Principle
From the second link,
“As noted, the accuracy of any measurement is limited by the wavelength of light illuminating the electron. Therefore, in principle, one can determine the position as accurately as one wishes by using light of very high frequency, or short wave-lengths. However, the collision between such high energy photons of light with the extremely small electron causes the momentum of the electron to be disturbed. Thus, increasing the energy of the light (and increasing the accuracy of the electron’s position measurement), increases such a deviation in momentum.”
That sounds analogous to what Andrew was writing.
Uh, the second part I forgot to paste ‘cuz I’m dumb:
“It is important to mention that The Heisenberg Principle should not be confused with the observer effect. The observer effect is generally accepted to mean that the act of observing a system will influence that which is being observed. While this is important in understanding the Heisenberg Uncertainty Principle, the two are not interchangeable. The error in such thinking can be explained using the wave-particle duality of electromagnetic waves, an idea first proposed by Louis de Broglie. Wave-particle duality asserts that any energy exhibits both particle- and wave-like behavior. As a consequence, in quantum mechanics, a particle cannot have both a definite position and momentum. Thus, the limitations described by Heisenberg are a natural occurrence and have nothing to do with any limitations of the observational system.”
Both of those agree with what I’m claiming. I’m not saying that observer effects don’t exist, just that they are different from the uncertainty principle and Andrew is talking about the former.
People *are* waves because everything is a wave, but wavelike properties and thus the HUP are pretty much irrelevant to social science and even to physics at these scales.
The irrelevance of quantum mechanics on macroscopic scales is not an obstacle for creative minds decided to rebuild social science on top of it.
See for example “Liberating Sociology: From Newtonian Toward Quantum Imaginations: Volume 1: Unriddling the Quantum Enigma” and “Quantum Social Science”.
The former is published by the Omar Khayyam Center for Integrative Research in Utopia, Mysticism, Science (Utopystics), the latter by Cambridge University Press.
“In this case, the news article reports on a company that ‘tries to address this shortcoming by constantly gathering data from small local internet forums, social media companies and messaging apps to determine public sentiment.’ I’m not sure how ‘public sentiment’ is defined here; they seem to be measuring something, but I’m not quite sure what it is.”
If you look at what is happening for instance on X/Twitter, these so-called SOCMINT techniques are worthless at best and harmful at worst (unless you are studying disinformation, propaganda, media manipulation, bot farms, trolls, scams, deep fakes, or something alike). Of course, you also have highly imbalanced samples as in many countries not many people use X/Twitter, only the elderly is using Facebook, youngsters use TikTok, and so on and so forth. Etc.
Quote from above: “This happens a lot in social science, and science in general. We have something of importance that we’d like to measure but is not clearly defined, we then measure something related to it, and then we have to think about what we’ve learned.”
I would like to take this opportunity to refer to one of my favorite papers I have come across during the time I attempted to usefully contribute to Psychological Science by reading and writing and publishing manuscripts.
The following quote is from a paper titled “The Lost Tools of Learning” by Sayers from 1947 (!), but perhaps is still very relevant today in light of all the possible sup-optimal reasoning and debating in society at large, and academia and science in particular:
“Have you ever, in listening to a debate among adult and presumably responsible people, been fretted by the extraordinary inability of the average debater to speak to the question, or to meet and refute the arguments of speakers on the other side? Or have you ever pondered upon the extremely high incidence of irrelevant matter which crops up at committee-meetings, and upon the very great rarity of persons capable of acting as chairmen of committees? And when you think of this, and think that most of our public
affairs are settled by debates and committees, have you ever felt a certain sinking of the heart?
Have you ever followed a discussion in the newspapers or elsewhere and noticed how frequently writers fail to define the terms they use? Or how often, if one man does define his terms, another will assume in his reply that he was using the terms in precisely the opposite sense to that in which he has already defined them?”