Skip to content
 

How to think scientifically about scientists’ proposals for fixing science

I kinda like this little article which I wrote a couple years ago while on the train from the airport. It will appear in the journal Socius. Here’s how it begins:

Science is in crisis. Any doubt about this status has surely been been dispelled by the loud assurances to the contrary by various authority figures who are deeply invested in the current system and have written things such as, “Psychology is not in crisis, contrary to popular rumor . . . Crisis or no crisis, the field develops consensus about the most valuable insights . . . National panels will convene and caution scientists, reviewers, and editors to uphold standards.” (Fiske, Schacter, and Taylor, 2016). When leaders go to that much trouble to insist there is no problem, it’s only natural for outsiders to worry.

The present article is being written for a sociology journal, which is appropriate for two reasons. First, sociology includes the study of institutions and communities; modern science is both an institution and a community, and as such it would be of interest to me as a citizen and a political scientist, even beyond my direct involvement as a practicing researcher. Second, sociology has a tradition of questioning; it is a field from whose luminaries I hope never to hear platitudes such as “Crisis or no crisis, the field develops consensus about the most valuable insights.” Sociology, like statistics and political science, is inherently accepting of uncertainty and variation. Following Karl Popper, Thomas Kuhn, Imre Lakatos, and Deborah Mayo, we cheerfully build our theories as tall and broad as we can, in the full awareness that reality will knock them down. We know that one of the key purposes of data analysis is to “kill our darlings,” and we also know that the more specific we make our models, the more we learn from their rejection. Structured modeling and thick description go together.

Just as we learn in a local way from our modeling failures, we can learn more globally from crises in entire subfields of science. When I say that the replication crisis is also an opportunity, this is more than a fortune-cookie cliche; it is also a recognition that when a group of people make a series of bad decisions, this motivates a search for what went wrong in their decision-making process.

A full discussion of the crisis in science would include three parts:

1. Evidence that science is indeed in crisis: at the very least, a series of examples of prominent products of mainstream science that were seriously flawed but still strongly promoted by the scientific community, and some evidence or at least speculation that such problems are prevalent enough to be worth our concern.

2. A discussion of what has gone wrong in the ideas and methods of scientific inquiry and in the process by which scientific claims are promoted and disseminated within the community and the larger society. This discussion could include specific concerns about statistical methods such as null hypothesis significance testing, and also institutional issues such as the increasing pressure on research to publish large numbers of articles.

3. Proposed solutions, which again range from research methods (for example, the suggestion to perform within-person, rather than between-person, comparisons wherever possible) to rules such as preregistration of hypotheses, to changes in the system of scientific publication and credit.

I and others have written enough on topics 1 and 2, and since this article has been solicited for a collection on Fixing Science, I’ll restrict my attention to topic 3: what to do about the problem?

I then continue:

If you’ve gone to the trouble to pick up (or click on) this volume in the first place, you’ve probably already seen, somewhere or another, most of the ideas I could possibly propose on how science should be fixed. My focus here will not be on the suggestions themselves but rather on what are our reasons for thinking these proposed innovations might be good ideas. The unfortunate paradox is that the very aspects of “junk science” that we so properly criticize—the reliance on indirect, highly variable measurements from nonrepresentative samples, open-ended data analysis, followed up by grandiose conclusions and emphatic policy recommendations drawn from questionable data— all seem to occur when we suggest our own improvements to the system. All our carefully-held principles seem to evaporate when our emotions get engaged. . . .

After some discussion of potential solutions, I conclude:

The foregoing review is intended to be thought provoking, but not nihilistic. One of the most important statistical lessons from the recent replication crisis is that certainty or even near-certainty is harder to come by then most of us had imagined. We need to make some decisions in any case, and as the saying goes, deciding to do nothing is itself a decision. Just as an anxious job-interview candidate might well decide to chill out with some deep breaths, full-body stretches, and a power pose, those of us within the scientific community have to make use of whatever ideas are nearby, in order to make the micro-decisions that, in the aggregate, drive much of the directions of science. And, when considering larger ideas, proposals for educational requirements or recommendations for new default statistical or research methods or reorganizations of the publishing system, we need to recognize that our decisions will necessarily rely much more on logic and theory than on direct empirical evidence. This suggests in turn that our reasoning be transparent and openly connected to the goals and theories that motivate and guide our attempts toward fixing science.

It’s fun, writing an article like this from first principles, with no position to defend, just trying to think things through.

8 Comments

  1. Peter Chapman says:

    Before anyone goes off making drastic changes to the way we do science we must look closely at the quality of the data we collect. Your section 3 (proposed solutions) lists a number of things that might need attention but doesn’t mention data. Many individuals who have studied and practised statistics over many years will have a deep understanding of the technical terms we use in statistics – e.g. error, significant, censored, random, sample – but many do not. In many years as an applied statistician I have seen so many howlers related to data that “poor data” must be a serious contender as a major contributor to the replication crisis. Some people who collect data have no idea what a sample is, let alone a random sample, so we shouldn’t expect them to be able to collect “good data”. We know, or can predict, from hypothesis testing theory what the replication crisis should look like. If there is a mismatch between what we are seeing and what we are predicting then either the theory is wrong, or we are applying it incorrectly. If the latter, I can’t believe that the problem derives from using the wrong statistical method on good data. It is much more likely to be the data that is at fault. If I am correct, the solution is simple – intensive training of data collectors. And this should not be academic style training in which you want to discriminate between the talented and the not so talented. It should be more like fire safety training where you want everyone to pass with flying colours.

    The problem with data as a potential solution to the replication crisis is that it is of no interest to the vast majority who are debating the issue: they are far more interested in promoting methodology.

  2. Terry says:

    Looks like stereotype threat is another casualty of the crisis. A meta-analysis of the literature finds little or no effect on high-stakes tests and finds strong evidence of publication bias. https://twitter.com/Russwarne/status/1183650756341092352 https://psycnet.apa.org/doiLanding?doi=10.1037%2Fapl0000420.

    This is a chance to test your innate BS detector. Did you ever think stereotype threat was a serious real-life effect? If so, you need to try to be less gullible in the future. (It always seemed like self-serving BS to me.)

  3. I don’t buy the reasoning in the first paragraph. Just because leaders in some area say there’s not a crisis doesn’t mean we should infer there is one. If we followed that inference strategy, we’d all be anti-vaccination and wear tinfoil hats to guard against cell tower radiation given how much health officials have warned us there’s not a problem in both cases.

    You say sociologists are good at embracing uncertainty. Does that mean sociologists better at statistics than other social scientists? I thought the field was primarily descriptive.

    • Andrew says:

      Bob:

      Regarding your first paragraph: I thought about this issue and addressed in my article, actually footnote 1:

      At this point a savvy critic might point to global-warming denialism and HIV/AIDS denialism as examples where the scientific consensus is to be trusted and where the dissidents are the crazies and the hacks. Without commenting on the specifics of these fields, I will just point out that the research leaders in those areas are not declaring a lack of crisis—far from it!—nor are they shilling for their “patterns of discovery.” Rather, the leaders in these fields have been raising the alarm for decades and have been actively pointing out inconsistencies in their theories and gaps in their understanding. Thus, I do not think that my recommendation to watch out when the experts tell you to calm down, implies blanket support for dissidents in all areas of science. One’s attitude toward dissidents should depend a bit on the openness to inquiry of the establishments from which they are dissenting.

      I guess the moral of the story is: Be careful when you excerpt an article, as not everyone will click through!

      • D Kane says:

        Rather, the leaders in these fields [including climate science] have been raising the alarm for decades and have been actively pointing out inconsistencies in their theories and gaps in their understanding.

        You really think that this is a fair summary of how some (many?) leaders in climate science behave? Folks like Michael Mann are not known for “pointing out inconsistencies in their theories.” Indeed, their primary claim to fame, at least from the point of view of this blog, is their refusal to share data and code.

  4. Michael J Lew says:

    It seems to me that the first sentence doesn’t belong to the rest. A crisis in “science” is a much broader claim than a crisis in sociology, and it seems to me that sociology is different from some other areas of scientific endeavour in ways that make it more prone to false or exaggerated claims.

  5. Dan F. says:

    What’s killing science and the rest of intellectual endeavors in universities is the hierarchical US university system and the star professor system at its top tier institutions. These overpaid celebrities are evidence of a decadent culture devoted to many things other than intellectual inquiry and skeptical thinking.

    Too much money, too many rankings.

Leave a Reply to D Kane