I’ve often appealed to “common sense” or “face validity” when considering unusual research claims. For example, the statement that single women during certain times of the month were 20 percentage points more likely to support Barack Obama, or the claim that losing an election for governor increases politicians’ lifespan by 5-10 years on average, or the claim that a subliminal smiley face flashed on a computer screen causes large changes in people’s attitudes on immigration, or the claim that attractive parents are 36% more likely to have girl babies . . . these claims violated common sense. Or, to put it another way, they violated my general understanding of voting, health, political attitudes, and human reproduction.
I often appeal to common sense, but that doesn’t mean that I think common sense is always correct or that we should defer to common sense. Rather, common sense represents some approximation of a prior distribution or existing model of the world. When our inferences contradict our expectations, that is noteworthy (in a chapter 6 of BDA sort of way), and we want to address this. It could be that addressing this will result in a revision of “common sense.” That’s fine, but if we do decide that our common sense was mistaken, I think we should make that statement explicitly. What bothers me is when people report findings that contradict common sense and don’t address the revision in understanding that would be required to accept that.
In each of the above-cited examples (all discussed at various times on this blog), there was a much more convincing alternative explanation for the claimed results, given some mixture of statistical errors and selection bias (p-hacking or forking paths). That’s not to say the claims are wrong (Who knows?? All things are possible!), but it does tell us that we don’t need to abandon our prior understanding of these things. If we want to abandon our earlier common-sense views, that would be a choice to be made, an affirmative statement that those earlier views are held so weakly that they can be toppled by little if any statistical evidence.
P.S. Perhaps relevant is this recent article by Mark Whiting and Duncan Watts, “A framework for quantifying individual and collective common sense.”
Working with surveys that attempt to uncover motivations and preferences for over 30 years, I have often told clients to follow the “90 percent rule.” In a well-constructed survey, about 90% of what you see should not be a surprise. The other 10% could be new knowledge which helps you make decisions in new situations. Any study where the main findings are discordant or shocking needs further investigation. For instance, many years ago a client with no experience did a study which concluded that divorced women aged 40 to 65 drank the most beer. It turned out they had asked participants to fill out detailed diaries about their beer drinking habits. And, yes, the heaviest beer drinkers based on hundreds of earlier studies, men aged 18 to 25, did a terrible job filling in and returning those diaries. The women were assiduous and relatively meticulous. So the “result” was the result of a flawed method. Subtler flaws, and sometimes even ones as blatant, often underlie most “shocking” study findings.
Andrew: its more a question of methods.
When the researcher is a physicist using the latest mass detction and measurement technology, we generally have high confidence in the measurement, even if the claim may be tenuous because of certain assumptions.
However, when a researcher makes claims based on statistical analyses about various issues in society, we can deploy our knowlege about both the technical veracity and common quality of statistical methods used in the social sciences. There is no need to bother with or consider our “priors” about the purported phenomenon. Since we know that trying to understand social phenomenon with statistics as practiced is akin to a blind person trying to put the puck in the net by taking a slap shot with a palm frond from center ice – e.g., the chances of whiffing are extremely high even for the best shooters – we should feel comfortable dismissing the results out of hand. The theoretical basis, the measurement methods and the analytical methods all have extremely low reliability (e.g., the shooter can’t accuratrely even sense his or her own manipulation of the palm frond, much less see the puck or the net). Thus, it’s the responsibility of the researcher to provide overwhelming evidence over an extended period of time (decades) to demonstrate that their claims are true. If they provide strong evidence repeatedly over, say, 25 years, we should consider considering their claims more carefully.
deb
> No doubt this makes your immune system happier.
I have a question about this. Do you have evidence to conclude that “an immune system” as a collective entity, becomes happier or less happy as a collective entity? Seems to me that your beliefs about Vitamins C, D, and Zinc would necessarily overlap with such a view. I’m not sure I believe it. My sense is that “the immune system” may actually be kind of a misconception. That what’s being conceptualized as a system is actually a lot of discrete elements that don’t likely respond to a single stimulus. One thing that complicates my view there is that there seems to be a distinct decline in immunity, broadly speaking, as people age. Or maybe not. I guess I should just Google or ask ChatGPT. But I figured it couldn’t hurt to ask.
Typically, people who study these things distinguish the innate immune system and the adaptive immune system, which are quite different. so, talking about ‘the’ immune system is a bit off. And, be careful of ‘strengthening’ your immune system; you may end up with an autoimmune disease.
Yeah! As a person with MS, whenever I see a product advertised as “strengthens your Immune system” I recoil and think ‘I really do not want that in my house!”
It will if you are actually deficient and take enough to fix the deficiency… or else we really don’t understand anything about vitamins.
More likely its like people putting random amounts of water on fires without directly looking at the effect on the fire.
IMO, the most revolutionary thing for health would be at home tests for the various essential vitamins and minerals like we have for glucose. Of course, even then it would only measure the amount in the blood and not necessarily in the other tissues where they do stuff…
We are still in the dark ages of medicine.
I expect that, as a political scientist who studies voting, Andrew had better intuitions for how much it varies than Kanazawa.
> It is taboo to say vaccines have side effects…
Wow. I was wondering why I’ve never heard anyone say that before. Because it’s taboo! I never would have guessed!
> You sound sarcastic
Moi?
Some friendly advice. Tone it down just a titch. Otherwise you may get asked to move over to Twitter.
deb:
When you reply to any comment, please click the “Reply” botton at the bottom left of the corresponding comment. You then don’t have to quote the whole comment you are referring to. Your comments will become much shorter and readable.
Also, it would help if you’d use an empty line or even quotation marks to distinguish what you quote from your own thoughts. As I understand it, this is basic courtesy in scientific writing.
” Good sense is dead, its child, science killed it to find out how it was made.”
From the novel, Innocence by Penelope Fitzgerald, the phrase is attributed to Antonio Gramsci..
You could replace good with common, but then it wouldn’t be a haiku.
Roger
Good piece of verse
Haiku is 5-7-5 syllables.
You have 4-7-4.
From haiku composer, and MD psychiatrist
Is it common sense to know what common sense is? Finally, some empirical evidence on the subject, courtesy of PNAS:
https://www.pnas.org/doi/10.1073/pnas.2309535121
A framework for quantifying individual and collective common sense
The notion of common sense is invoked so frequently in contexts as diverse as everyday conversation, political debates, and evaluations of artificial intelligence that its meaning might be surmised to be unproblematic. Surprisingly, however, neither the intrinsic properties of common sense knowledge (what makes a claim commonsensical) nor the degree to which it is shared by people (its “commonness”) have been characterized empirically. In this paper, we introduce an analytical framework for quantifying both these elements of common sense. First, we define the commonsensicality of individual claims and people in terms of the latter’s propensity to agree on the former and their awareness of one another’s agreement. Second, we formalize the commonness of common sense as a clique detection problem on a bipartite belief graph of people and claims, defining common sense as the fraction of claims shared by a fraction of people. Evaluating our framework on a dataset of raters evaluating diverse claims, we find that commonsensicality aligns most closely with plainly worded, fact-like statements about everyday physical reality. Psychometric attributes such as social perceptiveness influence individual common sense, but surprisingly demographic factors such as age or gender do not. Finally, we find that collective common sense is rare: At most, a small fraction of people agree on more than a small fraction of claims. Together, these results undercut universalistic beliefs about common sense and raise questions about its variability that are relevant both to human and artificial intelligence.
I know it when I see it, and that ain’t it. (to paraphrase Potter Stewart)
Older, but probably useful: Watts, Duncan J. 2013. Everything Is Obvious: How Common Sense Fails. London: Atlantic Books.
I do wonder that this research (Mark Whiting and Duncan Watts, “A framework for quantifying individual and collective common sense.”) is being cited, and its conclusions are being pondered while the real elephant in the room is the validity of the research. The unstated yet implied validity of their research is simply because they developed over 4000 statements to test with a large number of participants, that that somehow makes the conclusions valid. Instead 1) a review of the statements they used reveals a clear selection bias (defaulting to including easy to collect statements), mixing facts with “common sense” statements, and often confusing wording of the statements; and 2) clear bias in the participants to females and greater education.
However, two of the more interesting results they demonstrate (and undoubtably others have previously), is that 1) in some topic areas over others (religion, for example), people think that others believe things they personally do not, and 2) that groups tend to think alike. Unsurprising, and dare I say, “common sense.”