Skip to content
 

“Do you have any recommendations for useful priors when datasets are small?”

Someone who wishes to remain anonymous writes:

I just read your paper with Daniel Simpson and Michael Betancourt, The Prior Can Often Only Be Understood in the Context of the Likelihood, and I find it refreshing to read that “the practical utility of a prior distribution within a given analysis then depends critically on both how it interacts with the assumed probability model for the data in the context of the actual data that are observed.” I also welcome your comment about the importance of “data generating mechanism” because, for me, is akin to selecting the “appropriate” distribution for a given response. I always make the point to the people I’m working with that we need to consider the clinical, scientific, physical and engineering principles governing the underlying phenomenon that generates the data; e.g., forces are positive quantities, particles are counts, yield is bounded between 0 and 1.

You also talk about the “big data, and small signal revolution.” In industry, however, we face the opposite problem, our datasets are usually quite small. We may have a new product, for which we want to make some claims, and we may have only 4 observations. I do not consider myself a Bayesian, but I do believe that Bayesian methods can be very helpful in industrial situations. I also read your Prior Choice Recommendations but did not find anything specific about small sample sizes. Do you have any recommendations for useful priors when datasets are small?

My quick response is that when sample size is small, or measurements are noisy, or the underlying phenomenon has high variation, then the prior distribution will become more important.

So your question is a good one!

To continue, when priors are important, you’ll have to think harder about what real prior information is available.

One way to to is . . . and I’m sorry for being so predictable in my answer, but I’ll say it anyway . . . embed your problem in a multilevel model. You have a new product with just four observations. Fine. But this new product is the latest in a stream of products, so create a model of the underlying attributes of interests, given product characteristics and time.

Don’t think of your “prior” for a parameter as some distinct piece of information; think of it as the culmination of a group-level model.

Just like when we do Mister P: We don’t slap down separate priors for the 50 states, we set up a hierarchical model with state-level predictors, and this does the partial pooling more organically. So the choice of priors becomes something more familiar: the choice of predictors in a regression model, along with choices about how to set that predictive model up.

Even with a hierarchical model, you still might want to add priors on hyperparameters, but that’s something we do discuss a bit at that link.

11 Comments

  1. Matt Skaggs says:

    “I do believe that Bayesian methods can be very helpful in industrial situations.”

    A colleague described to me how my reliability prediction method could be improved using a Bayesian approach. I followed the argument and it certainly made sense, although we never followed though to see if it really helped.

    “Do you have any recommendations for useful priors when datasets are small?”

    I think the question is a bit ill-posed but clear enough in its meaning. The dataset of performance or reliability of the product is small with only four observations, but in most industrial situations, you have an established method for determining performance/reliability. So you might have a large dataset of “test set” performance for similar products that can be used for a prior. Between the test set engineers and the part designers, there should be a good understanding of test fidelity.

    The new product has novel features or it would not be new. Once you have established a reasonable prior for the test system, you can tweak it based upon the novel aspects. The engineers that designed the upgrades should be able to estimate how much the performance is expected to be enhanced (or not enhanced if the upgrades were only intended to reduce cost).

    Now if both the part and the evaluation (=test) methodology are new…I got nothing. Maybe the test technicians can work over the weekend and generate more observations!

    • Martha (Smith) says:

      “Maybe the test technicians can work over the weekend and generate more observations!”

      Depending on the situation, generating more observations may be a very expensive activity — for example, if gathering data requires alterations to the current industrial process (which might mean that the data gathering done so far is being done on weekends, or nights, or only when things are slow).

  2. Martha (Smith) says:

    I don’t have any personal experience with this, but much of my statistics education was from people with industrial experience. My understanding is that (as indicated in this thread) a typical industrial experiment was small, but that the practice was to do a series of small experiments, using what was found in the each experiment to shape successive experiments. Although this was not strictly Bayesian, it had in some sense the Bayesian spirit of making use of prior information (in this case, from prior experiments) in the design of successive experiments. So the first small experiments can perhaps give some reasonable bounds on variability; they can also give guidance in the design of the next experiment (e.g., avoid pitfalls of the earlier experiment), and it might point to areas on which you need to focus. In other words, you can’t expect a single experiment to give a definitive conclusion, but with successive learning from prior experiments, you can improve what you can get from the next experiment.

    • Keith O'Rourke says:

      > In other words, you can’t expect a single experiment to give a definitive conclusion, but with successive learning from prior experiments, you can improve what you can get from the next experiment.
      Yup!
      (Now how to we get the public to fully grasp this?)

  3. I thought that this commentary was quite revealing.

    Richard Smith: The most devastating critique of medicine since Medical Nemesis by Ivan Illich in 1975

    https://blogs.bmj.com/bmj/2019/02/13/richard-smith-most-devastating-critique-medicine-since-medical-nemesis-ivan-illich/?utm_campaign=shareaholic&utm_medium=twitter&utm_source=socialnetwork

    It relates to the point raised above:

    ‘I always make the point to the people I’m working with that we need to consider the clinical, scientific, physical and engineering principles governing the underlying phenomenon that generates the data; e.g., forces are positive quantities, particles are counts, yield is bounded between 0 and 1.”

  4. Anoneuoid says:

    we want to make some claims

    The problem is already evident here. If you already know what claims you want to make from the data then it is all pointless.

    The point of analyzing your data is to figure out what claims are supported, if any. The popular method is p-hacking of one kind or another, but I’d say fiddling with priors makes it even easier to “prove” whatever you want from stats (at least in terms of generating a favorable “headline number”).

Leave a Reply