What’s the purpose of mathematical modeling?

Peter Dorman writes:

I think this is an example of what good science journalism looks like. The description of modeling methods and even workflow is as accurate as it could be without getting beyond the technical background of its readership. Nice graphics! I like the discussion of the tradeoff between granularity in model design and propagated uncertainty and especially the prominence given to measurement and data availability issues. And the value of having multiple models working in parallel, related through a model intercomparison framework. Also, there is a lot of useful discussion of the problem of communicating model outputs and their dependence on model design, data and assumptions.

The only thing I found to be missing, and this is forgivable considering the purposes of the article, is the consideration that should be given to the choice of outcome metrics. In theory, pandemic modeling should be in support of public health, but up to now it has prioritized outcomes of interest to the health system. So: we get projections of deaths, hospitalizations and utilization of ICUs and respiratory support but little or nothing on symptoms not requiring hospitalization or persisting after it. No doubt this is due primarily to data availability, but as the article shows, the supply of data is, sooner or later, responsive to the demand for it. I fear that, in this and other countries, there is an implicit assumption that a primary goal of public health is to form a protective ring around institutionalized health resources. Of course, not overloading hospitals or unnecessarily burdening their care protocols is beneficial because of the services they provide, but the end purpose of modeling should be public health, no? Which means there should be a demand for data on the full spectrum of significant health outcomes of the pandemic and modeling effort to increase our understanding of them.

The article he’s pointing to is by Jordana Cepelewicz and begins:

The Hard Lessons of Modeling the Coronavirus Pandemic

In the fight against COVID-19, disease modelers have struggled with misunderstanding and misuse of their work. They have also come to realize how unready the state of modeling was for this pandemic. . . .

And it has this good quote:

Scientists — not just in epidemiology, but in physics, ecology, climatology, economics and every other field — don’t build models as oracles of the future. For them, a model “is just a way of understanding a particular process or a particular question we’re interested in,” Kucharski said, “and working through the logical implications of our assumptions.”

17 thoughts on “What’s the purpose of mathematical modeling?

  1. ‘For them, a model “is just a way of understanding a particular process or a particular question we’re interested in,” Kucharski said, “and working through the logical implications of our assumptions.”’

    Until the press release.

    • An interesting but totally unknowable question is what proportion of “mathematical models” for COVID were created in order to generate support for a specific set of policies or practices. I know in any field there will be scientists who are genuinely motivated by a desire to just add to our understanding of mechanisms that underlie certain phenomena. I’m not sure that desire for pure, theoretical knowledge was much of a factor in the publication of mathematical Jeremiads concerning the effects (or not) of various COVID mitigation policies.

      • I agree with your concern. The article did a good job of explaining the difficulties with modeling COVID (and, more generally, all models), but seemed to implicitly suggest that the modelers had pure intentions. I think many of the modeling efforts were complicit with various stakeholders – and, at least were riding the wave of publicity associated with any possible implications of COVID models. The motivations of modelers was not given the attention it should have received, in my opinion.

        • event if modellers have pure intention, there is a biased publication filter.
          Say If you are a COVID modelled inside governmental health organisation, government can decide which of your results to publish, so they can always claim they “listen to scientists”.

        • Somewhere in between “pure” intentions and “complicity” with stakeholders is an amorphous area that can kind of shade into either extreme depending on the thing being studied. I’m talking about a model whose creators aren’t necessarily doing political or policy advocacy but they do start out with WHAT THEY THINK IS GOING ON.

          That’s where the real quicksand lies for all kinds of modeling, not just COVID or other politically fraught topics. Where exactly do you draw the line being pure intentions in the most abstract, open-minded sense of going wherever the data leads you vs. pure intentions to “see if they data supports our hypothesis”. Once you set out to find support for your theory, you’re worryingly close to advocating for your theory, no?

        • I don’t think so at all. The question is what you do if your theory predicts things that aren’t right. Do you ditch the theory and move on, or do you go try to find some other data that will support it and try to keep people from testing your model against the other data etc?

          Hypothesizing mechanisms and checking to see if they predict things appropriately is what science is about.

        • “The question is what you do if your theory predicts things that aren’t right. ”

          Depends on what you mean by “predict”! :) Seems like an easy word to handle but when both the prediction and the data that would otherwise be used to verify the prediction have significant slop, then whether or not a “prediction” has succeeded or failed is open to question. It takes a *long* time to narrow down the hiding space that a committed opponent of a theory or hypothesis can fit into.

          I think that’s what Brent is referring to – the biased modeler can have their cake and eat it too. The reality is that very few models deliver definitive predictions that ultimately force the modeler to significantly change direction. That’s the nature of modelling with fuzzy data.
          papers.

  2. The purpose of mathematical modeling is to formalize sensible but ill-defined questions into answerable questions, sometimes sacrificing sense for answers, and too often misunderstood to be answering the original questions.

  3. My current kick at the can.

    It is always the case that given we have no direct access to reality, reality must be represented abstractly in our heads.

    Given that we must think about reality using abstractions, we can only notice aspects of those abstractions – just the models we make.

    Math can be defined as formalizing ways to notice aspects of abstractions or models.

    Models take elements and relations among them in the represented world [that produced the data] and map them onto elements and relations in the representing world [probability world]. Models are thinking tools.

    This requires transporting what repeatedly happens given a probability model to what reality just happened to produce this time.

  4. The point about why scientists build models is somewhat thought-provoking. I don’t think I agree with it.

    One of my current contracts involves modeling what portions of PG&E’s electric grid are most likely to be involved in ‘ignitions’, i.e. in starting a fire. This will be used to help PG&E decide what portions of their infrastructure to ‘harden’, where to do increased tree-trimming, and so on. In a sense the model is indeed an ‘oracle’: the primary interest is in its predictions: these are the specific locations where the risk is highest, here are some other locations where the risk is low, and so on.. However, maybe I would agree that to the extent that our interest is in the model’s predictions, the model is not ‘science.’

    Although the primary interest is in the model’s predictions, there is also interest in related questions, such as what additional information would allow the model to be improved, and how the risk of a fire caused by a tree branch falling on electric wires varies with wind speed and tree species and so on. These are definitely science.

    When climatologists try to create improved models for forecasting climate change, they’re doing science. And although the model is “a way of understanding a particular process or a particular question we’re interested in,’ that isn’t ALL it is: it is ALSO an ‘oracle’ that we consult to find out what will happen in the future. I don’t like the word ‘oracle’ because of its black-box implications, but my point is that in many cases a model serves multiple purposes: we use it to understand a particular process, yes, but we also use it to make forecasts.

    • Even your fire model is not intended to be an oracle: the desired outcome is that some additional action is taken so that the predictions of your model do not happen. This chain of logic is probably so obvious to you that you didn’t consider it worth spilling ink, but based on the national conversation over the last year and a half (and longer, with climate models) it is worth pointing out. We use models to make conditional predictions, not unconditional ones.

      • But Kevin – isn’t the conditional piece – If our model does a good job of capturing the causal mechanisms, functional forms and strength of relationships of fire patterns then it will make good predictions? Isn’t the quality of the prediction conditioned primarily on how close the model is to the ‘truth’?

        • The point is that if we predict “if you do nothing you’ll start 100 fires”, and then you go off and do something… and only 13 fires are started… it’s not a failure of the model.

          the prediction “you’ll start 100 fires” is not of the form “unconditionally, you will start 100 fires” it’s of the form “if you do nothing, you’ll start 100 fires”.

          This was ignored over and over again when it came to COVID. People predicted things like “if we do nothing there will be 3 million deaths in UK by June 2020” then there weren’t 3 million deaths in the UK by June 2020 and it was taken as “evidence they didn’t know what they were talking about”. It was only evidence that “if we do nothing” wasn’t true. we did things.

          (note, numbers just made up here, I didn’t look up the specific prediction quantities)

    • Couldn’t agree more, Phil. The idea that we can divorce the understanding of the world contained in a model from its ability to make predictions to independent data is misguided. All of our understanding of the world is contained in our models of the world – and the only way to estimate how much understanding those models hold is through the predictive ability of the models

Leave a Reply

Your email address will not be published. Required fields are marked *