Skip to content
 

Using statistics to make the world a better place?

In a recent discussion involving our frustration with crap research, Daniel Lakeland wrote:

I [Lakeland] really do worry about a world in which social and institutional and similar effects keep us plugging away at a certain kind of cargo-cult science that produces lots of publishable papers and makes it easier to get funding for projects that don’t really promise to give us fundamental and predictive models that can drive real improvements in people’s lives.

It’s sort of a “it’s 2014, where’s my flying car?” attitude I know, but I’d be satisfied with a lot of things other than flying cars, such as:

1) real, effective solutions to antibiotic resistant organisms
2) cures for cystic fibrosis
3) reducing the effect of heart disease on people under age 75 by 30%
4) understanding major causes of “the obesity epidemic” in a real detailed way and finding effective ways to reverse it.
5) Being able to regenerate organic replacement joint components instead of titanium hip implants etc
6) Growing replacement kidneys
7) A more significantly more effective and long lasting pertussis vaccine

Is the way we are doing science today going to provide any or all of these things in the next 30 years? What are some similar order of magnitude things that it has provided since 1980 using current “modern” methods and funding priorities, publication priorities, tenure systems, and so forth?

I want to consider this question, difficult as it is. It does seem that slow progress is being made in our conceptual understanding, at least in the sense that new paradigms are developing in different sciences. In sociology there’s an increased interest in network effects, in medicine it seems that a lot of things are being now understood in terms of the microbiome, in large-scale biology there’s a focus on the fetal environment, in microbiology we’ve been hearing a lot about the rapid evolution of viruses, etc. It seems to me, as an outsider in these fields, that the concepts I’ve mentioned have been around for awhile, but it’s only recently that they’ve been moved to the center of discussion. And this, in turn, seems to be the result of millions of research studies on many thousands of topics, which collectively ruled out earlier favored explanations. By revealing areas where the old paradigms failed, all this research made room for new, more difficult, paradigms to be taken seriously.

It’s posterior predictive checking, on a larger scale. A chicken is an egg’s way of constructing another egg, and empirical research is a scientific theory’s way of uncovering the theory’s flaws.

It’s harder to make such clear statements about statistics or engineering or computer science, as these are essentially tools in the service of science rather than being the object of study themselves.

And there are fields where new paradigms don’t seem so apparent. Consider three areas with which I’m somewhat familiar: political science, psychology, and economics. In political science, I see persistent difficulties in integrating different perspectives coming from the studies of public opinion, institutions, and political maneuvering. It really feels like we’re not seeing the whole elephant at once. And I include my own research as an example of this incomplete perspective. Psychology seems to be undergoing a reforming process, in which various unsuccessful paradigms such as embodied cognition are being rejected, with no clear unification of the cognitive and behavioral approaches. Similarly in economics, although there it seems worse in that various incomplete perspectives are taken by their proponents as being all-encompassing.

The point of that previous paragraph is not to pass judgment on these different social science fields; it’s just my impression that they are in various stages of reconstruction and reform. From my outsider’s perspective, biology and medicine seem to be in better shape, philosophically speaking, in that there seems to be more of a recognition that traditionally dominant paradigms miss a lot.

How does statistics fit into all this? Statistics can (potentially) do a lot:
– Guidance in data collection and the assessment of measurements. And recall that “data collection” is not just about how to collect a random sample or assign treatments in an experiment; it also includes considerations of what to measure and how to measure it. For example, if you are interested in measuring the most fecund days of a woman’s monthly cycle, it makes sense to check out the medical literature on the topic.
– Methods for calibrating variation by comparing to models of randomness. This is where I think that statistical significance and p-values fit in: not as a way to make scientific discoveries (“p less than .05 so we get published in the tabloids!”) but as a measuring stick when interpreting observed comparisons and variation.
– Tools for combining information. That to me is the most general way to think of “inference,” and it encompasses all sorts of things, from classical “iid” models to more complicated approaches including all sorts of time series and spatial analyses, multilevel models for partial pooling, regularization methods that allow users to ramp up the amount of data they can include in their models, and Bayesian inference for balancing uncertainties.
– Methods for checking fit, for revealing the aspects of data that are not well explained by our models. To me this includes all of exploratory data analysis, which is about learning the unexpected. Or, to step back slightly, exploratory data analysis is about putting us in a situation in which we are able to learn the unexpected. And, remember, “unexpected” = “not expected” = “something not predicted by our model,” where this “model” may be implicit.

Again, statistics is in the service of science, and I see statistics as a way of organizing science rather than as a way of making scientific discovery.

Sometimes we can perform a statistical analysis that seems to take us all the way from data to model to scientific conclusion—for example, this cool age-period-cohort model with Yair—but, even there, I think most of the real scientific heavy lifting is coming from existing substantive theories; the statistics is more of a way of rearranging the data or, as Dan Kahan would put it, of adjudicating between competing hypotheses or underlying models of reality.

31 Comments

  1. BenK says:

    I work on a scientific interface with Lakeland’s objections in mind. Places like 1,2,4 and 7.

    I think we can safely say that in areas of biodiversity or genetics – that is, areas in contact with molecular biology, sequencing, etc – and those touched by mass spectroscopy, imaging, microscopy, remote sensing – we are beginning to make massive progress in unlearning incredible volumes of falsehood. This is going to be dreadfully painful and is causing major problems in the ‘gearbox’ of translational research.

    Unlearning is not something that regulations and arts entertain. Refinement, they cope with very well.

    As a result, the science is in a tight place. We would hope for the ‘customers’ to advocate for the science they need – but they don’t know they need it. It doesn’t look relevant to them. This makes it difficult to fund the research; the timelines are wrong, the regulations are contrary, there are competing pressures to do the wrong things more efficiently. Industry doesn’t want it, physicians don’t want it, and scientists who have been co-opted into short-term payoffs know where their bread is buttered.

    The science will muddle on (unless society really grinds to a halt in the meantime). The pull toward the microbiome in infectious disease and vaccinology is already too strong for medicine to ultimately resist. Same with non-linear toxicology, better models of cancer, human diversity in metabolism/nutrition, and a number of the other unlearnings. The current system isn’t perfect, but it isn’t static either. I don’t think that we are collectively pursuing ultimately pointless incremental research – though we could certainly do much better (I’m looking at you, NASA A-3 test stand).

    • I’m glad to hear you say this. I think the unlearning is a serious issue. One of my main complaints honestly is the lack of theory in many sciences. Theoretical Biology and Theoretical Medicine in particular aren’t even “a thing”. Oh sure, lots of people have theories in Medicine, in fact we have the term “evidence based medicine” because so much of historical medicine is unchecked-theory. I don’t think that means theory is useless though, it means we need really *good* theorists, people who go beyond the kind of thing we have historically like “osteoporosis is the loss of calcium from the bones, so calcium in the diet should help”. In fact, all the progress in osteoporosis drugs is about targeting the balance between osteoclasts (bone-eating cells) and osteoblasts (bone-depositing cells) which ultimately comes from building theories that are more connected to reality.

      • Corey says:

        Theoretical biology and theoretical medicine will never be things because by its very nature biology (by which I mean the thing itself, not the field of study) is a historically contingent mess.

        • That doesn’t mean there aren’t lots of opportunities for applying theory to biological questions. The goal of “theoretical biology” or “theoretical medicine” wouldn’t be to come up with some kind of universal “standard model of biological mechanisms” to compliment the physics “standard model of particles and interactions”, it’d be more like taking substantive questions in biology, and then bringing physics, chemistry, biochemistry, observational biological facts, and soforth together to build models that actually address the question in a rational way. This kind of thing DOES get done, but it’s a patchwork, not something that people can really specialize in doing.

          Example from my wife’s postdoc research: there is a model for limb growth proposed ages ago about a “progress zone”, essentially it’s a brick-laying analogy that says the longer a cell stays near the end of the outgrowing limb, the more likely it is to decide internally that it should participate in building a distal structure (like an elbow joint, or a carpal bone) vs a proximal structure (like a shoulder joint or a femur). There are all kinds of problems with this model, both epistemologically, and biochemically. On the other hand, there are known signalling molecules that have known diffusivities and binding energies, and are connected in known pathways, and soforth. If you believe in such a model, it should have all kinds of physical consequences, and perturbations would be predicted to produce phenotypes that are different from perturbations of an alternative system…

          Biofilms are common in certain types of bacterial cultures associated with all sorts of pathogenicity (from tooth plaque to postoperative infections in hip replacements, to chronic surgery and antibiotic resistant sinusitis). Lots of things have been observed to go on in biofilms. They have a unique kind of geometry, surface chemistry, chemical environment… there are a variety of mechanisms by which they could conceivably cause pathogenicity. Which ones are dominant in common cases? Is the diffusivity decrease of the thick matrix responsible for reduced antibiotic concentration? Is the observed DNA sharing responsible for more rapid onset of antibiotic resistance? Is there a surface-affinity which affects the distribution of drugs?? In each of these various cases, what would be the consequences for bacterial infection *in situ*? Observing colonies in a dish can be quite misleading.

          • Corey says:

            When you write that this is “not something that people can really specialize in doing”, do you mean it’s not possible to specialize in it in principle, or do you mean that the incentives aren’t set up correctly (or do you mean something else entirely)?

            • I mean the incentives and/or social structures aren’t set up, it’s not a role that is recognized and exists in academia or pharma or etc, there are very few appropriate educational tracks and etc.

              • hjk says:

                I’m confused – bioengineering, biophysics,,systems biology and mathematical biology all exist as academic fields and are arguably more fundable than ever and with more training programs than ever. Especially when there are possible applications to medicine. Pharma is also more interested in ‘modelers’ than ever.

              • hjk: my impression of bioengineering is that it’s not so interested in biology per se as much as figuring out how to do things with biological systems. More, “engineering with biology” than “applying engineering knowledge to understand biological systems in-situ”. I admit I may have a mis-impression.

                My impression of biophysics is it’s more of an “excuse for doing physics”, studying the mechanics of molecules and principles of stat-mech as they apply to receptors and soforth, with the emphasis on discovering interesting bits of physics, not so much as discovering how physics can help understand biological systems. (again, just impressions from tidbits of spectating mostly)

                Systems biology is perhaps the most fundamentally similar to what I was talking about, but my impression is it’s not very interdisciplinary, more of “biologists who are taking things a little farther in terms of looking for interactions between pathways” and things like that.

                mathematical biology I associate with more of an “excuse for doing math”

                I don’t have deep connections to any of these fields. I certainly know a fair amount about some people who are doing biology and the bioinformatics type of “systems biology”.

  2. Keith O'Rourke says:

    > Tools for combining information.
    The phrase “combining information” often misleads people, it appraising all the information (I once called auditing scientific process) and then jointly processing the _common_ information which might be a simple weighted combination. Interestingly, appraising all the information in different studies was initially ignored in genetics and only recently taken up with some enthusiasm.

    Ramsey, also made the point that what he termed the logic of discovery (induction) could only be assessed inductively – has it seemed to work well in recent practices.

  3. numeric says:

    Since you are familiar with political science (I’m not giving you economics and psychology), what would be your list of noteworthy accomplishments over the last thirty years (and a list of what you would like to see done). Perhaps you’ve already done this and can share a link.

    • Andrew says:

      Numeric:

      You ask for my “list of noteworthy accomplishments over the last thirty years” in political science. My answer is: compared to biology and medicine, I got nuthin. Here’s what I wrote in 2008:

      Social scientists may know a lot, but they haven’t done that much. Compare to physical and biological sciences and engineering. Research in these areas has given us H-bombs, chemical fertilizers, laptop computers, vaccinations, ziplock bags, etc. etc. And social science has given us . . . what? An unbiased estimate of the incumbency advantage? The discovery of “nonattitudes”? A clever way of auctioning radio frequencies? The discovery that sumo wrestlers cheat? Not much “news you can use,” I’d say. I guess there’s been some work in epidemiology that’s been useful. Certainly some interesting things [such as] polls which give us a sense of the distribution of opinions on lots of issues—but I don’t think this comes close to comparing to the achievements of the so-called hard sciences.

      • Rahul says:

        Could that be made into a good argument for pushing less resources into the social sciences? Or more? I wonder….

      • numeric says:

        I’m pretty much in agreement with you. But I would argue it is the epistomological closure of the social sciences (in political science, debates on party id, then rational choice models, etc–in economics, business cycle models that don’t allow for exogenous events, in psychology, a search for ESP!). For example, social science made a huge contribution in the 1960’s with the Coleman report (for those who are too young to remember, this report essentially said that black achievement in schools was approximately equal to white achievement when the blacks were in white schools–this formed the basis of busing with its corresponding white flight and reaction). The Coleman report was a classic case of confusing correlation with causation (the act of moving African-Americans to white schools through busing was not the same as whatever processes put those few that were there in the 1960’s)–Coleman later repudiated busing, incidentally.

        The point is, the social sciences could have been consequential had there been an emphasis on real world problems, such as alleviation of poverty. This was not, of course, as interesting as mathematical models with no real-world applicability. In fact, most social science interventions are not conceived of or implemented with academia (think of the Gates foundation). I would argue that a lot of the dishonesty in our politics (mostly, but not all, from the right) comes about because political scientists/economists repeat obvious falsehoods (the non effect of the stimulus, the lack of influence of race on elections) and those that don’t do not call them on it. This is (very slowly) changing, perhaps–Ornstein on Congressional obstructionism, Krugman on economic policy, but the main problem is that most of the economics/political science profession is stuck in the mindset/attitudes of the 70’s. Tenure ensures dinosaurs will be around forever–AI with logical consistency testing will be required to eliminate this, and this will ensure that no progress will be made in the social sciences until AI can actually conceptualize and consistency-check theories.

        • Andrew says:

          Numeric:

          1. I agree with much of what you write. But I disagree with your statement that researchers have viewed working on real-world problems as not “as interesting as mathematical models with no real-world applicability.” I think people have worked on mathematical models etc because that’s easier. It’s natural when doing science to attack the easy problems first and then develop a general understanding, step by step.

          2. Even without tenure, I have a feeling that dinosaurs will be around forever. There are lots of dinosaurs with the political savvy to stay around in an organization.

        • krippendorf says:

          “The point is, the social sciences could have been consequential had there been an emphasis on real world problems, such as alleviation of poverty. This was not, of course, as interesting as mathematical models with no real-world applicability. In fact, most social science interventions are not conceived of or implemented with academia…”

          There is, and has been for most of the modern (post-1950s) era, a focus on “real world problems” in sociology. Jim Coleman was a sociologist. His near contemporaries, Daniel Patrick Moynihan and Sir Michael Young were academic sociologists who studied poverty before they became public officials. In the 1980s and 1990s, you couldn’t pick up an issue of one of the flagship journals in sociology without encountering empirically-based papers on poverty, social mobility, racial or gender inequalities in education or wages, changing family structures, etc etc.

          The impact of all this sociological research on “real world problems” on public policy is certainly debatable. But, please don’t assume that because topics weren’t fashionable in economics or political science until relatively recently, they weren’t studied in “the social sciences.”

    • WB says:

      Ideal-point estimation is, I think, a major contribution by political science. Poole and Rosenthal’s NOMINATE scores have given us an ideological history of the Congress, showing us that partisan polarization has varied considerably over the many decades. That the two congressional parties are more polarized today than they have been since Reconstruction tells us something fundamentally important about U.S. politics. At a minimum, it tells us whether we should expect bipartisan compromises on major issues or whether we should believe candidate promises of a new, post-partisan era. In short, a clear sense of the ideological divisions in Washington can help voters appraise claims by candidates, pundits, and other political actors. And by featuring this kind of research, websites like The Monkey Cage are slowly bringing NOMINATE and similar scaling methods into mainstream political discussion.

      Moreover, McCarty, Poole, and Rosenthal have shown, convincingly, the strong correlation between income inequality and polarization in Congress. This finding has since launched considerable research into why democracy, and the U.S. system in particular, struggles to slow the rising income gap. The answers, so far, are depressing and suggest that things will get much, much worse before they get better.

      Is this research as noteworthy as developing vaccines or improving fertilizers? On an obvious level, no. But diagnosing the political problems associated with inequality–one of the core problems that the United States faces–may help identify viable reform strategies. You can’t develop a cure if you don’t what causes the affliction.

  4. Andrew: it made my day to have you call out that comment in an end-of-year retrospective kind of way. I have a blog post from just the other day that I think is relevant to this, and it’s got a Gelmanesque pop-culture image-reference to boot.

    The point of that post was to call out the role that model building should play in science and statistics. I think, especially in biology and medicine, but also in lots of the call-out cases in social sciences you’ve come up with over the last few years, there just isn’t any substantive theory, at best there’s a surface theory (ie. “evolution should favor pinker colors during peak fertility”). That’s inherently vague and not particularly predictive. There’s not much in the way of mechanism, and even if we could somehow discover that it’s true, so what?

    Contrast that with some of the great work you’ve done and discussed in BDA, pharmacokinetics models that actually predict internal concentrations of toxins, radon risk models that can actually be used to help people make decisions about controlling their lung cancer risk, or well-switching models that can help people understand how to educate and inform villagers about poisons in their water supply.

    In all of those cases we have a model that makes predictions, gives us actionable information, and unlocks at least some of the internal keys to how something works, or how to make better decisions in the future.

    Statisticians can’t be experts in all the substantive sciences they come in contact with, but they can be more than just experts on applying probability theory. They should think about alternative model structures, about looking for mechanistic explanations, about whether the models they are building answer a very narrow question, or elucidate some entire swath of questions all at once.

    My wife, a biologist, likes to say “biologists use statistics like a drunkard uses a lamp-post, for support rather than illumination” and she means this in a derogatory (or at least self-deprecating) way. It’s no good for statistics as a profession to be seen by scientists as a kind of seat-belt and ABS brakes to keep them from getting hurt, or as a kind of one-two punch to apply to critics when they get a low p value. Statistics needs to interface more directly with science by stepping up as strong leaders or at least collaborators in model-building, and I especially think that the Bayesian perspective is critical to the model-building process.

    • Rahul says:

      I like the way you have framed your question. More people ought to ask whether a given study is worth funding.

      It has become almost unacceptable to ask whether a certain area of work has become over-funded. Such questions are met with great hostility.

      e.g. So much theoretical work on Quantum Computing gets published. I’ve even seen papers about the impact of QC on e-commerce. In practice, a real, scalable computer is not even close, & the best QC we can build today does something trivial like factor the number 24. It is not even clear that we can ever build a QC. Even if you want to work on QC why not focus on the precise bottleneck i.e. decoherence & noise.

      Basically how does one address the question of relative allocation of research resources to various problems?

  5. Steve Sailer says:

    Check out George Hawley’s recent work in political science.

  6. Fernando says:

    “reducing the effect of heart disease on people under age 75 by 30%”

    I’m surprised this has not been solved. Almost every week I read in the newspapers about a new epidemiological study “showing” how eating [pick food] every day reduces your risk by at least that much.

    If only more people read newspapers I am sure rates would come down fast. For real.

  7. Student says:

    What do you mean by the term “embodied cognition”? The research program in the field of social psychology, or a somewhat broader term used in the field of cognitive science?

  8. John Mashey says:

    It may be that I don’t know the others, but for two of these, it seems major contributory causes are reasonably well-understood by biology/medicine, but obscured by politics.

    3) Heart disease: some if this might be related to cigarettes/nicotine, rather well documented, with increasing clarity in Surgeon General reports. The corporate/political efforts to obscure that are well documented in the 80M+ pages in the Legacy Tobacco Document Library at UCSF.

    4) Obesity: thank Earl Butz, High-Fructose Corn Syrup and the sugar-beet growers.
    Follow UCSF researcher @RobertLustigMD.
    There is almost certainly a similar set of sugar documents paralleling the tobacco documents, but only the first few have gotten loose, sometimes from litigation between the corn refiners and sugar beet producers.
    A lot of this is politics as well.

    Anyway, these seem qualitatively different from at least some of the others.

  9. question says:

    “major contributory causes are reasonably well-understood by biology/medicine”

    If this were true there would be someone out there with a theory making precise predictions. I doubt this is the case, but would love to see an example showing I am wrong. I think people, both in and outside, greatly overestimate our understanding of biology/medicine. Most of that “knowledge” is based on p<0.05 type research. A key aspect of this popular approach to research is that once there is some narrative people find appealing, little effort is expended into investigating alternative explanations or procedural artifacts.

    Just think, the primary paradigm people use for explanatory mechanisms is still signaling "pathways", when we know it is more like a network. How accurate can our knowledge of these systems possibly be?

  10. John Mashey says:

    Heart disease: US Surgeon General 2014, Chapter 8.
    While heart disease has many contributors, smoking (or even just nicotine) are major contributory factors,
    One can find more details of the mechanisms in some of hte talks at UCSF’s <a href="Billion Lives symposia.
    2013:
    Even Brief Secondhand Smoke Exposure Disrupts Arterial Function, Matthew Springer, Associate Professor of Medicine, Cardiology (1h:36m:48)
    2014:
    Smoke-Induced Heart Attacks: The Underlying Biology, Peter Ganz, Professor & Chief, Division of Cardiology,
    I attended thsoe talks, but won’t try to summarize: you’re much better watching the videos.

    Sugar and obesity (/diabeytes):
    Quantity of Sugar in Food Supply Linked to Diabetes Rates
    Google: sugar consumption obesity rates

    In both cases:
    a) The statistical evidence of linkage is strong.
    b) Understanding of exact mechanisms may vary, but one doesn’t have to have perfect understanding to think:
    stop smoking if at all possible, and if nicotine-addicted, try to minimize it and watch out for all the added sugar.

    • Smoking and sugar have been well understood problems for decades though. I mean cigarettes were called “coffin nails” even before the 1940’s, and at least over-consumption of calories (which sugar contains a lot of) is understood to be bad for you.

      My point was more about coming up with fundamental understandings of heart disease risk that could be applied to the rest of us (at least where I am in California, smokers are a small minority, most middle class people know about and make some effort to eat at least moderately health diets, etc)

      as far as I know, even statins haven’t been shown to actually reduce heart disease risk, only cholesterol levels etc. the causal link is still missing, and the side effects can be significant.

      • John Mashey says:

        Sorry, broken link, that was: Billion Lives symposium above.

        “We don’t know everything, but we do know some things” fits these cases as well as many others.

        For instance, there was a long a focus on lung cancer, caused mostly by the tar in cigarettes, and especially in lawsuits, since few other things cause that. It’s taken a while to sort out the effects of nicotine on heart disease/strokes, whether from direct smoking or secondhand smoke, and they are actually getting to mechanisms, like relationship between nicotine and angiogenesis. One hopes that research in that direction may help udnerstand underlying mechanisms that occur in people without nicotine exposure.

  11. Andrew, could you back this claim up:

    “Psychology seems to be undergoing a reforming process, in which various unsuccessful paradigms such as embodied cognition are being rejected, with no clear unification of the cognitive and behavioral approaches”

    What’s the basis for this statement?

  12. […] way of Andrew Gelman, we come across a very good question posed by Daniel […]

Leave a Reply