We’ll ask an expert, or even a student, to “tell me what you know” about some topic. But now I’m thinking it makes more sense to ask people to tell us what they don’t know.
Why? Consider your understanding of a particular topic to be divided into three parts:
1. What you know.
2. What you don’t know.
3. What you don’t know you don’t know.
If you ask someone about 1, you get some sense of the boundary between 1 and 2.
But if you ask someone about 2, you implicitly get a lot of 1, you get a sense of the boundary between 1 and 2, and you get a sense of the boundary between 2 and 3.
As my very rational friend Ginger says: More information is good.
This is my single biggest problem with the way so much operates: people hide what they don’t know and they toward dense jargon as a method of obfuscation. I think this process is internalized and operates mostly at a partially conscious level.
Don Rumsfeld is the authority on this topic :) :
http://youtu.be/GiPe1OiKQuk
I recall how he was excoriated in the press for this. People (specifically, talking heads whose job is to create controversy) became unglued because they weren’t able to make sense of it. Not that I’m necessarily a fan of his, but I thought this was actually a quite astute statement, for exactly the reasons that Andrew mentioned.
On the upside of controversy, you get things like this:
http://www.stuffedpenguin.com/rumsfeld/documents/TheUnknownclip160k_000.mp3
The following advice is from Lewis Thomas,the great medical educator, theorist, and essayist.
I suggest that the introductory courses in science, at all levels from grade school through college, be radically revised. Leave the fundamentals, the so-called basics, aside for a while, and concentrate the attention of all students on the things that are not known.
Lewis Thomas
Read more at http://www.brainyquote.com/quotes/authors/l/lewis_thomas_2.html#HRsMyp7hkRKQAd8T.99
Oooh, I hate Lewis Thomas’s writings, which to me epitomize a showy “humanism” beloved of my high school English teachers. I much prefer Oliver Sacks, who fills his essays with information rather than merely banging on a point over and over.
Well, I have read almost nothing by Lewis Thomas. The quote is about the limit of my acquaintance with his work, and I like it and the idea behind it.
(I have read only one book by Sacks. It was OK but made no great impression on me)
The stuff you don’t know you don’t know, is either your mistakes with the stuff you do know, or stuff you don’t even have a pathway to knowing. Usually, the stuff you don’t know you don’t know is what you need to ask others about.
Andrew:
Can you tell us what we don’t know about Statistics? Might be worth a blog-post?
I slightly disagree: I think asking about 1) also helps with calibrating 3):
a) They may not know they don’t know something, but nobody knows. That gives little information.
b) They be making an implicit assumption without thinking about, but you know and can tell them.
c) (This is the case where asking 1) helps), they may explicitly say they know something, but you have strong evidence they are wrong. That is good information, especially if you give them the evidence and it is rejected.
Example: software startup seeks funding from Sand Hill Road venture capital firm.
VC: who’s your competition?
Startup:
1) We know of X and Y, but we’re doing something different.
2) There might be somebody back in Boston or Hew York doing this, but we don’t know well enough.
3) Of course, the VC may have seen a similar proposal a week before, or even be far along in funding somebody.*
If VC asks startup:
How sure are you that X and Y are the only competitors around here?
Good answer: we’ve looked, couldn’t find any, but from long experience there might be stealth startups or internal efforts, but we’ve already done good development on a shoestring, have track record for good execution, etc.
Bad answer: we’ve looked hard and we are absolutely certain no one else is doing this.
* This happens quite often.
Andrew, do you think there might be potential for your idea to overlap / update the following..?
https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
The real nuance here is what does it mean to know? It can only be understood in context of relative comparisons, is and can never be static (as what we think we know changes across time and tasks, heck just being asked any of the three has different certainity confidence effects from asking), and both the asker and respondent may suffer from a lack of effective communication in conferring why/how they know, not know, or not think they know. Knowing is a collective postulation with subtleties in expression, conference, of degrees of applicability. Thus the answer to the first two is always the upper and lower bound limits of what all parties participating in the exploratory endeavor, with the answer to three likely residing in those who are not participating. Knowing anything belongs to no one of any of us.
Reminds me of this book http://www.amazon.com/The-Encyclopedia-Ignorance-Everything-wanted/dp/0671790870 that was certainly influential with a much younger me…
Thanks, I am finding it helpful in thinking through some issues raised here http://statmodeling.stat.columbia.edu/2015/01/24/teach-hypothesis-testing/
You’ve left out a potential 4, “unknown knowns”: things that people assume about a topic, but aren’t consciously aware of assuming, either because some piece of knowledge has become thoroughly internalized, or because that ‘knowledge’ is an unconscious presumption. Asking about 2 is perhaps also likely to reveal the boundaries of 4. (Idea is from Žižek.)
Pingback: Handy statistical lexicon - Statistical Modeling, Causal Inference, and Social Science