Skip to content
 

Rob Kass: “The truth of a theory is contingent on both our state of knowledge and the purposes to which it will be put.”

Here’s a presentation, Exaggerated Claims Undermine Science by Ignoring the Scientific Method, by Rob Kass, a statistician who over the years has done a lot of interesting work on statistical theory and applications, especially in neuroscience. A few years ago, we discussed Kass’s thoughts on statistical pragmatism. And here’s a discussion of a couple of papers by Kass and Brad Efron, which may be the first time I quoted Hal Stern that “the big divide in statistics is not between Bayesians and non-Bayesians but rather between modelers and non-modelers.” (That’s similar to the other Hal Stern quote that “what’s important in a statistical method is not what it does with the data but what data it uses.” I guess Hal has a certain style with his aphorisms.)

Rob argues that the way that a big problem in science communication is that the way that science is presented—as a set of known facts—does not match the give-and-take of the scientific process.

True science is collaborative or kaleidoscopic in many ways: many models of the world, many sources of data, many techniques of measurement, many researchers (including yourself at different times). The exciting process of scientific discovery is not well captured either by dry textbooks (at one extreme) or the scientist-as-hero framework of Freakonomics and Ted talks.

In his talk, Kass goes into details on a social neuroscience example, discussing among other things the often misunderstood distinction between correlation and causation. As someone who’s done a lot of work in survey research, I’d also like to emphasize that correlation does not even imply correlation. Also, a minor thing: Kass discusses how a regression “controls” for variables. I prefer to say “adjust for.” But that’s all minor. Overall I recommend Rob’s talk in that it connects general issues of science to more specific questions about what is statistics.

14 Comments

  1. Phil says:

    What’s important in an Hal Stern aphorism is not what Hal Stern says about an issue, it’s what the aphorism says about Hal Stern.

  2. Olav says:

    The slogan in the title strikes me as an unhelpful conflation of epistemology and metaphysics. It seems better to say: whether and to what extent we can determine if a theory is true is contingent on our state of knowledge. But whether a theory is in fact true does not depend on our state of knowledge.

    Kass endorses Box’s slogan that “all models are false, but some are useful.” This slogan sounds good, but it raises the question of what it means for a scientific model to be “useful.” Arguably, the goal of science is to uncover truths, or partial or approximate truths, about the world. Hence a scientific model is scientifically useful to the extent that it accomplishes this goal. A model is not useful–qua scientific model–if it just makes you famous or rich or boosts your confidence, although a model that has these latter properties is clearly useful in a different sense. So, to make it more specific and accurate, perhaps we should rephrase Box’s slogan so that it says, “all models are false, but some are partly or approximately true or say something true about the world.” But that’s not as snappy.

    On the other hand, Kass seems to propose that we redefine “truth” to mean “provides excellent explanations of a certain set of results.” Hence, the goal of modeling is (apparently) not to uncover truths (in the traditional sense), but rather to come up with excellent explanations. But this proposal just raises the question of what counts as an “excellent explanation.” Can an explanation be excellent even if it doesn’t describe the world in a way that approximately corresponds with the way things actually are (i.e. even if the explanation isn’t approximately true in the traditional sense of “true”)? If the answer is “no”, well then approximate truth (in the traditional sense of “truth”) is the goal of scientific modeling after all, and redefining the word “truth” is not very helpful. (By the way, there’s a literature in the philosophy of science that deals with the question of whether a good explanation must be true.)

    • a naive philospher of science says:

      the goal of science is something that philosophers of science have debated and continue to debate, so you might want to check on that before arguing for the goal of science to be the establishment of truths or partial/approximate truths as if these were out there waiting for us. I would appreciate an example of one of those truths discovered by science that was not later deemed to be partially incorrect, and therefore approximate, or false, by means of it not being useful anymore to explain phenomena (and of course do not count mathematics). I would side with Box and Bayesians for that matter because we cannot explain anything without a model, and yet the model, by its own nature, is that, a model, not a replica, and serves a purpose: to explain a tiny little corner of our complex world, not to discover things that are out there in the wild (unless you are thinking more about paleontologists and like-minded science people who classify things, that is different, that is purely discovery, but you still need a model that tells you where to look for things)

      • Olav says:

        Yes, there’s a philosophical debate on what the goal of science is, so I appended “arguably” to what I said. That said, what I said is weak enough that most participants to that debate could agree with it.

        As for whether scientific truths are discovered or invented—strictly speaking truth is a relationship that obtains between a theory and the world (and yes, I’m assuming a particular view of truth here—there’s a philosophical debate on what as well). The world is certainly not invented, so truths are at least in part discovered, but whether theories are invented or discovered is a subtler question.

        • jim says:

          Phenomena and the relationships among them are discovered. The theory in which we encompass them is an invented conceptual wrapper that metamorphoses as our understanding of the phenomena and relationships changes.

        • a naive philospher of science (again) says:

          propositions in mathematics can be TRUE or FALSE. But was proposition P always T before proven to be T? Or did it became T after the it was proven? Theories are more complex than propositions because they involve many of those types of propositions, and outside of mathematics, and perhaps physics at some levels (though i consistently ignored physics after high-school, to my shame), do establish some of those propositions as T or F, and build up from that. In the realm of social sciences, or even biology, no theory is ever proven TRUE. Not ever ever ever in the history of science. If not, I would please ask you to point to one single one case. Just one, a tiny example of a scientific theory that has remain truth.
          on a side note, that type of thinking, of truths out there, is what lies behind many current debates. people dont understand how our ideas evolve and change because the stuff we thought were truths are found to be incomplete and utterly mistaken (see gender as binary, race as biological, homosexuality as disease, genes as causes of disease, etc. etc. etc.)
          we want to explain the world, not find truths, and those explanations are always forever and ever incomplete. thats good, otherwise we wont have jobs in the future when science is finally complete!

    • Olav: There a fair amount of discussion on what you raise here by Andrew and I at http://www.stat.columbia.edu/~gelman/research/unpublished/Amalgamating6.pdf

      One quote ” We add here that we strive for more than just not being frustrated by resistance from reality; rather, we want our findings and
      claims that aim at truth to be “beliefs which succeed for reasons connected to the way things are” (Misak, 2016).

  3. It is very thoughtful and informative, but I believe unavoidably? awkward.

    > “The truth of a theory is contingent on both our state of knowledge and the purposes to which it will be put.”
    Whoa! – why not replace “truth” with purposefulness (pragmatic usefulness) rather the overly vague and high invested with differing underlying philosophical stances “truth”.

    So in the end, a vague presentation on the problems and risks of vagueness in scientific discourse.

    (To avoid such vagueness, I have been focusing math as diagrammatic reasoning in order to present probability models as diagrams that as abstract diagrams can be learned about to any degree of accuracy using simulation. So making the abstractness of the Theoretical side precise to hopefully make it clear it should not be confused with the world which will always remain vague.)

  4. Pritish Patil says:

    Just wanted to give an “original” link of for the video with Closed Captions.

    https://www.youtube.com/watch?v=P2pfapZ5no0

  5. Hey Friends, Andrew, Joshua may be interested.

    I arranged Dr. Michael Mina to be one of three keynotes here in DC. Joshua and others can ask questions.

    DATE: Thursday, September 17, 2020, at 7 pm

    It will be featured on the Ward 3 Democrats DC Facebook page, with a Zoom link to be posted as the event date nears.

    FACEBOOK LINK: https://www.facebook.com/Ward3Democrats/

    Hope you can join and ask SUPER questions, as you all are so knowledgable. Or if you have suggestions for questions, PLEASE email me: sameeradaniels@gmail.com

    • Andrew,

      Thank you so much. I really appreciate your posting it. Nor would I have been offended if you hadn’t posted the event notice. After all, this is your blog. It’s that you guys are the most analytical and creative questioners I’ve come across.
      Cheers

Leave a Reply