Wanted: a catalog of the sorts of problems that can and can’t be solved using statistics.

Anton Treialt writes:

I have been playing around with Bayes for a while now and the first feelings about the results are somewhat mixed. For my type of problems – small number of observations and wide priors linked to some economic effect – the confidence bands are typically really wide. Which is kind of expected as the number of observations is small and priors is wide. When I am looking at the outcomes I wonder whether it will really enhance any decisions people have to make based on this analysis.

Which leads me to a suggestion for a post on your blog.

I was reading this post, the one that began, “Many years ago, when I was a baby economist, a fight broke out in my firm between two economists. . . .”

It would be very interesting to learn from someone who has seen it all, which kind of problems cannot likely be “solved” using statistics and the answer will depend mostly on beliefs. Maybe it is because of latent variables, maybe because of small number of observations, maybe something else. Perhaps someone has already come up with some sort of classification of problems that are very difficult to tackle using stats?

I don’t know, but it’s a good question. The dividing line between “can be solved” and “cannot be solved” is blurry, but there does seem to be something to this idea.

15 thoughts on “Wanted: a catalog of the sorts of problems that can and can’t be solved using statistics.

  1. Isn’t the question of what problems can or cannot be “solved” with statistics akin to the question of what p-value designates an effect as “real?” I think both suffer from binary thinking. All evidence is only suggestive – some more convincing than others. And, “convincing” when confronted with ideology makes the statistics even more subject to prior beliefs. The dividing line between can and cannot be solved is blurry and must remain so. But there are degrees of blurriness, and the point at which a person is willing to conclude something will vary across people, particularly if they have strong beliefs about the issue.

    I don’t think there is anything controversial in what I have said (we’ll see, as I am confident others will let me know if there is). But that raises a real question of what role statistical analysis can/should play in decision making. My intuitive feeling is that the noisier the evidence, the smaller the role for statistics, but I’m not sure about that – and I think that might be worth discussing.

    • The noisier the evidence the MORE role for statistics. If you have a curve plus a tiny bit of noise then even simple least squares will identify the relationship. If you have a curve plus a lot of noise and the noise is of multiple types and there is some theoretical reason to believe there are change points where the curve behavior radically changes and there is a data collection issue where definitions changed at some point in history and there is a similar curve in multiple countries but the exchange rates make that curve complicated to measure…. Now you’re talking Bayesian statistics.

      As for the OP, the really wide intervals is the important result. The point of good statistical analysis is to tell you how much you know about the way things are, if you don’t know much and the stats tell you you have precise information that’s a BUG not a feature.

    • Dale:

      Statistics is useful in the phase transition between problems that are so easy that statistics is not needed, and problems that are so hard that statistics won’t help. If the signal is strong enough, you don’t need statistics to demonstrate it; if the signal is weak enough, statistics won’t find it for you.

      Put this way, the role of statistics seems pretty minor—and it is! Statistics is not as important as science and engineering.

      For example, it’s great to statistical quality control to improve the efficiency of a automobile production line. But that’s all trivial compared to the effort involved in inventing the car, refining petroleum, etc. Statistics helps on the margins.

      That said, we do spend some time at these phase transitions. The easy problems get solved right away, then we get interested in the harder problems. We didn’t need statistics to know that national elections are close; we did need statistics to figure out Red State Blue State.

      • Reading what Andrew and Daniel just said – and both were well stated – it does raise a question. Is there a relationship between the degree of noise and the difficulty of the problem? Daniel claims that statistics is more important the noisier the data and Andrew says that statistics won’t help if the signal is weak enough. I don’t think they are really in disagreement here, but is it possible to tease out a bit more what makes a problem difficult and what role the noise in the data plays in that?

        • I think we need to distinguish between the solution of two different problems… Problem 1 is you want to get “the answer” to a real world problem. Ie. You want an answer sufficiently precise that the uncertainty is either irrelevant or at least very manageable.

          Problem 2 is you have a model of the world and some data, and you want to find out what those two imply about some aspects of the world (could be predictions or parameters or whatever).

          Problem 2 is the problem Bayesian statistical analysis solves. Problem 1 can only be solved by both having a good model and a lot of data collection and some luck that the problem itself has a stable sort of answer and doesn’t require constantly updating models to account for new phenomena.

          The problem is people keep thinking that stats solves problem 1, this is particularly true because NHST basically tells you to test your model and then if you have the right p value to “act as if” you had solved the problem and know the answer… This is just wrong in every way.

          Bayesian stats is an add on to a model, it’s a way of investigating the implications. It specifically gives you samples of plausible states of the world, all of them are plausible not just one. When you don’t know much, Big uncertainty **is the solution** of the problem Bayes tackles. It doesn’t lie to you and tell you to pretend you know more than you do.

        • Daniel
          I just don’t get your distinction between problem 1 and problem 2. You say 1 is about a real world problem and in 2 you want to find out something about some aspects of the world – these sound the same to me. Also, I think the Bayesian issue is a red herring in terms of a potential catalog of problems that statistics can or cannot help with. To the extent that you think Bayesian models are better than non-Bayes approaches, this should expand the set of problems that statistics can help with – but I don’t think it turns a problem that is not amenable to statistical analysis into one that is (perhaps that is debatable, but it feels somewhat off the point to me).

          I am starting to think that noisy data may be sufficient to make a problem difficult, but not necessary. Often the data is not very noisy, but it either is too little or measures the wrong things. In such cases the problem becomes difficult. I also think noisy data makes problems difficult. But the question is what is the proper role of statistics in such problems? I kind of agree with your statement that statistics becomes more important in such situations (after all, what is the alternative?). But I also agree with Andrew’s statement that some problems are so hard that statistics won’t help.

          I’ve been involved in many regulatory proceedings where I would say the problem is difficult (most often because the data is insufficient for the task), but the statistics seems important but cannot/does not resolve anything. On the other hand, I would say that the Harvard Admissions case is an example of a difficult problem where the statistical analysis helped – if you read the judge’s decision about the competing econometric models, I think the analysis was helpful with a difficult problem.

          What is the difference? The regulatory decisions were highly politicized and regulatory commissions are highly politicized. In the Harvard case, while the issues were certainly politicized, the judge appears to have distanced themselves from the politics sufficiently to examine the evidence carefully. Unfortunately, legal cases appear to be getting more and more politicized.

          So, I am wondering if the focus on difficult problems (raised by Treialt) should be focused more on difficult decisions. When there are competing values and poor decision making processes, then statistics doesn’t really help (although it provides a good living for consultants and politicians).

        • The distinction I was trying to make is between the substantive problem (eg. How many people have melanoma in the US at the moment?) and the inference problem (given my model of the world and the causes of melanoma and the population distribution geographically and my understanding of how this melanoma survey was carried out and the raw data from the survey, how much do I know about how many people have melanoma in the US today?)

          The first question is what people always want the answer to… What is the true value?

          Stats doesn’t give you that, it gives you the answer to the second question. And Bayesian methods are not a red herring here because they are the only ones that even answer the second question.

          Frequentist stats answer a different question still that most people don’t care about at all… It’s something like… Given a model of the survey sampling procedure as pure random number generator what is the probability of this survey sampling procedure giving me a dataset where the summary statistic is more different from some given value than the observed statistic was…

      • “…Put this way, the role of statistics seems pretty minor—and it is! Statistics is not as important as science and engineering…”

        Sci. and eng. utilize some of the Physics (the easy part) to obtain good enough results, practical for our world.
        However, deep down at the level of a single electron, it’s all about probability distributions and that’s Statistics for you. I actually think that Stats/Probability are more important than anything.

        “…if the signal is weak enough, statistics won’t find it for you…”

        If stats doesn’t find it, nothing else will. What is there to replace stats/prob. when the signal is too weak?

  2. I think the major question here is how much can or cannot be identified/known from the available data. Obviously this is not a binary issue; small sample sizes only allow imprecise statements, but how precise is precise enough depends on the situation.

    But there are other problems, for example dependence can only be detected from data if it is roughly assumed how dependence works, for example along the time line or within individuals/groups in mixed effects models. In cluster analysis, the data can’t tell you what the “true clusters” are, because this requires a decision about what actually defines a cluster. There are different incompatible definitions and the data will not tell us which one to choose. Causality directions can only be identified in certain situations etc.

    • I agree with this, but would state it slightly differently. Most of the statistical approaches require that the data generating and sampling systems can be adequately captured or otherwise dealt with in the statistical model. Sometimes ‘adequate’ can be attained pretty easily, but often it is impossible.

      It is a shame that the aphorism coined by George Box, “All models are false, but some are useful” is so often used as an excuse to assume usefulness rather than a reason to worry about the quality of the recipe of model, data, and inference. A lot more thought and discussion should be spent on justification of models and the extrapolation of model-bound inferences into the real world.

      • +1

        Also, the continuation of the Box quote illuminates precisely this point: “Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.”

  3. Our lack of ability to foresee which complex problems will get solved and which won’t extends to statistics and machine learning. I used to lecture to computer science students on why machine language translation and speech recognition were so difficult, and yet we now take the solutions for granted, mostly.

    So, here is a massive challenge in healthcare – how to usefully model something that continues to evolve? (There’s plenty of data, although it is difficult to get at.)

    An example:

    Recently, I had the best lipid profile in my adult life. I asked my doctor, “Why?” He said, “Cardio and no restaurant food.” He didn’t know that I gave up the gym early in the pandemic, so no strength training, and my sole exercise was hiking around my neighboring hills, so, yes, plenty of (low intensity) cardio. (I checked, and the literature shows that cardio is an excellent way to lower cholesterol.) Of course, the lack of strength training (I’m old) causes other problems, but that is another story.

    He could guess that I cut restaurant dining to almost zero during the pandemic.

    But, as the readers of this blog understand, this is far from a controlled experiment!

    Some time ago, I explored trying to model metabolic syndrome. It’s very complicated, though common; I gave up, lacking the large amount of data that analysis would require. Now ask how much harder it would be to analyze metabolic syndrome data across the pandemic? Would the output of a future exercise monitor – my watch – be associated with my improved lipid profile? This is an example of something that is potentially analyzable, but not realistically analyzable today, or in the near future.

    During the same visit, my doctor ordered a hepatitis C test. He said the CDC now recommends that adults get the test; he said it was because we can now cure it. Since, as he put it (this was a Berkeley physician), I wasn’t a sex worker or a porn star, he had not ordered such a test beforehand. I had not seen the CDC guideline, so the roll-out must have been understated and subtle.

    I will ask my infectious disease contacts about this because this change will make data analysis even more difficult than usual. (It’s hard because hepatitis C can go unnoticed, and then be transmitted, without the test.)

    So, statistics and machine learning can, in principle, take on data evolving over time, but I don’t see “best practices” for such analyses as well developed.

    Considerable progress HAS been made analyzing COVID outcomes, and the effect, or non-effect, of proposed therapies – and this in the face of mutations of the virus. But, larger public health analyses, such as predicting metabolic syndrome, seem to be beyond current analysis. So, the problem goes into the “Difficult” category, for the foreseeable future.

    Upon reflection, I hope I did not assert during my long ago lectures that machine language translation and speech recognition were intrinsically unsolvable. I hope I made clear that just because we could not SEE our way to over-coming these challenges that they could not be overcome.

    So, I think the latter distinction is one that needs to be made regarding problems we label as difficult.

Leave a Reply

Your email address will not be published. Required fields are marked *