45 thoughts on “Ecologists’ endless quest for automatic inference”

1. I find this post very confusing. You start with the idea of training/test data splits but that would seem to have little to do with “automatic inference,” which seems like a strawman to me. No doubt you will find support on this blog for trashing the idea of automatic inference – and I don’t seek to support it. Many on this blog insist on specifying a data generating process and that makes sense to me where it is feasible. In social science, I think it is often elusive, and the best you can hope for is using variables that theory suggests should be relevant. Apparently the situation is ecology is somewhat similar (though there are more biological, chemical, and geological processes that seem more amenable to a scientific approach). When there are many potential variables, then your suggestion of picking a few that make sense seems suboptimal – various automated feature selection procedures would seem superior to me. Of course, the resulting model should make sense, and I’m a big proponent of the importance of subject matter knowledge. So, if your strawman is using the computer to automatically select features and then automatically select the model, then I don’t think you will get any objections from me or anyone else.

   But if you are lamenting the use of these automated processes, then I’m on a different page. I wouldn’t substitute them for a more scientific deductive approach, but I think they are complements, particularly as the amount of data (number of variables) grows large. I also think that averaging a variety of machine learning models can be a reasonable thing to do. In any case, I keep finding performance of different machine learning models on the same data is more similar than different, despite the different underlying methodologies.

   Returning to the theme of splitting the data into training/validation/test data sets, I no longer see any justification for failing to do so. I’ve had this discussion with various LLMs and also real statisticians, and the consistent answer I get why much published work (outside of data science) fails to validate their models is that change is slow. I have found no justification for failing to validate your models. The closest I hear is that much academic research is interested in inference, not prediction. Thus, the emphasis on p-values and the absence of validation. As far as I can tell, that position makes no sense. Even if you are only interested in inference, on what basis can you trust your model? A low p-value from a model that can’t predict well seems fairly useless to me.

   One thing you say about machine learning I don’t understand at all. You say such methods give “(good) predictions for out-of-sample data.” I don’t see how they can do that at all. They are all limited by their training data, so out-of-sample performance should not be expected. Indeed, I thought that is why validation is done to begin with. Even then, out of sample performance is still constrained by the limits of the data available to begin with. So, what do you mean by saying they give good predictions?

   Reply ↓

   • Dale –

     I don’t understand your position. I do not think that the author was using any strawmen here. The entire purpose of splitting data into training/validation/test sets is to assess the out of sample predictive performance of a model (i.e. external validity). Out of sample prediction is a fundamentally different task than mechanistic inference which has it’s own algorithms to test for external validity (i.e. retrodictive model checks). Inference and prediction are extremely likely to select completely different sets of covariates or even model structures. If I understand Lizzie’s blog post correctly, this is the danger in using algorithms designed to assess the validity of *predictive* models (which tend to fall under what may be viewed as “automatic inference” algorithms like AIC, BIC, and… cross validation with training/test/validation sets) to select a model intended for mechanistic inference.

     Prediction-oriented model selection is highly sensitive to the idiosyncrasies of the training data. A model that performs well in prediction may do so by leveraging spurious associations or regularities that are unhelpful—or even misleading—for the purpose of understanding underlying mechanisms. This is particularly problematic in observational data, where bias from uncontrolled confounding or post-treatment conditioning can distort inference, even if prediction error remains low. Conversely, model selection / building for inference should involve tool that help guide variable selection and model structure based on known or hypothesized causal relationships, rather than predictive performance alone. In these cases, cross validation makes no sense and thus there is zero use for a “validation” data set.

     Lastly, I want to push back on your critique of Lizzie’s comments about “good out-of-sample predictions.” Predictive models are useful only insofar as they generalize beyond the training set—i.e., they must have good out-of-sample performance… But I think you’re conflating the concepts of out-of-sample prediction and scope of inference. Consider the example of spatial Gaussian processes, or geostatistical kriging models. These models perform well in out-of-sample prediction within the spatial domain of the data. Their performance degrades as predictions are made further from observed data—this is a scope of inference limitation, not a failure of out-of-sample generalization per se. The distinction is important: out-of-sample prediction refers to performance on withheld data from the same population; scope of inference refers to whether the target of prediction lies within the same data-generating process.

     Miles

     Reply ↓

     • Miles
       I don’t understand most of what you way. You say machine learning models “must have good out-of-sample performance.” Surely that is desirable, but why must they have this. It depends on the data and the models. I’ve built many machine learning models that don’t have good performance on validation or test data, so I don’t get your or Lizzie’s use of “must.” If you are just expressing a desirable property, or a required property for a “good model,” then it is just the terminology that I find confusing. But if you mean this is a necessary condition of such models, I just don’t get it.

       I also don’t get your statement that inference and prediction “are extremely likely to select completely different sets of covariates or even model structures.” That is certainly possible, but from my experience I haven’t seen such marked differences. You state that inference should have tools for feature selection that make validation of “zero use.” Absent a clear data generating process (such as physical relationships that perhaps exist in some physical science contexts), I think most social science models have tools for feature selection that make validation necessary, not useless. If I am modeling crime rates across a range of cities, what can I rely on for feature selection that would make validation useless?

       There is perhaps a problem of language here. You use the terms “automatic inference” and “mechanistic inference” and Lizzie does as well. I don’t know these as well-defined terms, so I am interpreting them as I imagine they might be used – hence my statement about “strawman” construction. If these terms are unambiguously defined, then my statements are off-base. I’d appreciate clear definitions of these terms in that case. My statements referred to practices of automatically accepting a model based on a specific metric (whether it be AIC, BIC, p-values, or something else). I think we all agree that such practices are not recommended and can lead to all sorts of problems. But I consider that a sort of strawman argument against machine learning models. Such automatic model selection is not a unique characteristic of machine learning models – it can just as easily be malpracticed with any inferential model, I think.

       Reply ↓

       • Dale –

         Thanks for your response. I apologize if my language is unclear. To be succint: Machine learning models “must” have good OOS prediction because otherwise they are useless for OOS prediction… and OOS prediction is strictly the only thing a model selected for OOS prediction is useful for.

         “Modeling crime rates across a range of cities” looks very different if all we care about is predicting crime rates across a range of cities. This model will look very different if, instead, we wish to understand what *causes* crime rates. This is at the heart of Lizzie’s blog post here (I think!) as well as my comment above.

         You ask “what can I rely on for feature selection”? The answer to this questions depends on the modeling goal. If the goal is simply prediction, then cross-validation and related algorithms are exactly what we should use! If the goal is to causally interpret the coefficients, then we may consider DAGs or similar structural causal modeling frameworks. If a system of study (e.g. social determinants of crime rates in your example above) is so complex that we can’t even draw out a reasonable DAG, then I can almost promise that a predictive model’s coefficients are hopelessly biased. This means that the model will be (hopefully) good at OOS prediction, but interpreting the coefficients of the model is misguided. The results of cross validation, step-wise model selection, or feature importance plots absolutely should not be interpreted causally.

         By models meant for “mechanistic” inference, I mean that we intend to interpret coefficients of the fitted model causally. A reference to clarify what language I am using and where I’m coming from is Chapter 7 from Richard McElreath’s text and the associated lecture here
         https://www.youtube.com/watch?v=odGAAJDlgp8

         Miles

       • Miles
         Thanks for the reply. It will take awhile for me to digest the video, but in the meantime I have a question. I agree that if we want to interpret a model causally, then observational data is not ideal and machine learning models will require a great deal of care (I won’t go so far as to say such interpretation is impossible). But if we only have observational data, then do you believe it is possible to interpret any model causally without doing validation? In other words, validation is not sufficient for causal interpretation, but is it necessary? My own thinking is that it is necessary but not sufficient.

       • Dale —

         Thanks for the thoughtful question! I think you’re getting at what is, in my own understanding, the crux of the issue. My position is that causal inference with observation data is tenuous but possible under specific sets of assumptions. The issue with using validation (here I am using validation as in the language of this paper https://esajournals.onlinelibrary.wiley.com/doi/10.1002/ecy.3336) is that the process of selecting a model based on it’s prediction error on withheld data tends to favor confounded models and thus the resulting parameters can’t be interpreted causally. To reword this, if we introduce OOS prediction and validation into our model selection decision process, we are likely to select an incorrect model if our goal is causal interpretation.

         The tenuousness of causal model selection with data generated from anything except a perfect RCT / experiment is exactly why most (any?) “automatic inference” algorithms are dangerous, IMHO.

         Cheers,

       • Miles
         I am clearly having trouble with the terminology. In the article you link, I see the following two statements in successive paragraphs:

         “Inference does not require validation on independent data, because the goal is to determine whether or not a particular data set provides sufficiently strong evidence for a hypothesis.”

         “Statistical inference from a single data set is only one part of the process of developing scientific knowledge. It assesses the reliability of statements about a particular set of data obtained by particular methods under particular conditions. Any conclusions obtained via statistical inference thus require replication and validation across a range of conditions before they are accepted as scientific fact.”

         These seem contradictory to me. Also, the discussion of whether there are two, three, or more competing models just seems like strange thinking to me. Perhaps it is specific to ecology and my different disciplinary background is causing me confusion. But I would think that we almost always have multiple plausible models to consider. And your reference to a “perfect RCT” strikes me as unrealistic as well. Most RCTs (particularly in medicine where costs rise rapidly with sample size) share some of the same problems as observational data. In the simplest case, the control group and treatment group will not be exactly the same aside from the treatment. Even though the assignment is random, the differences between the groups may still matter and that makes the analysis look somewhat like observational data (the selection is random in one case but not the other, but in both cases the differences between the groups cannot be ignored).

         What I really don’t understand is the following:

         “if we introduce OOS prediction and validation into our model selection decision process, we are likely to select an incorrect model if our goal is causal interpretation.”

         I agree that there will be problems, but I don’t see why validation makes it any more likely to select an incorrect model if the goal is causal interpretation. How does failure to validate make it less likely to select an incorrect model?

       • Miles
         One more addition. I read the whole article you linked and it is quite interesting. But I think the key point of our divergent views is illustrated in this statement:

         “Repeating the analysis using only the covariates featured in the inference example led to a much larger loss of predictive accuracy. These results are reminders that an understanding of process does not always translate directly into improved predictive skill.”

         I fully accept the caution in the final sentence. But in the case they are looking at, I don’t see how there inference necessarily reflects a better understanding of process. It was a multiple regression model with an interaction effect. Absent any validation, what is the basis for concluding that it is a better understanding of process? The only justification I see is based on the p-values. As Anoneuoid keeps reminding me, the coefficients in such a model cannot be trusted to be meaningful. If we don’t validate that model, then why should we attach any more legitimacy to that understanding than the one that results from a better predictive model?

         I don’t mean that the best predictive model is equivalent to the best causal model. But in the real and imperfect world, I don’t see how eschewing validation can be justified on the basis that we are looking for an inferential analysis. Again, I see validation as necessary but not sufficient for both purposes (inference and prediction).

2. Excellent post Lizzie, I heartily agree. In my experience, nothing beats a sustained exposure to Richard McElreaths lectures and materials on causal inference and DAGs to clarify the difference between scientific inference and predictive model selection/evaluation. Once you see it, you can’t unsee it :)

   Reply ↓

3. I wish you the best of luck with any attempt to lure people towards any process that requires thought and judgement and away from “automatic inference”. I have for years tried to understand why people think of p-values as merely a component of the automated decision process of a Neyman–Pearsonian hypothesis test rather than a (neo-)Fisherian index of evidence that can be used to inform a thoughtful inference. I suspect that the turn the crank recipe of an all-or-none dichotomising hypothesis test allows users to absolve themselves of any responsibility for the conclusions (“computer says no!”), but almost certainly ignorance and laziness also play a role. Presumably the same applies to some measure in the case of machine learning.

   Reply ↓

4. This post couldn’t have come at a better time, because I’ve been thinking about these topics for a few months now.
   From the little I know, tools like WAIC, AIC, LOO compare models on the basis of out-of-sample prediction, but in reality it seems like almost everyone does a mix of causal inference and prediction in their papers, so maybe that’s the problem there? Miles seems to agree with me on this, up above.
   Anyway, “too many climate variables” seems to be everyone’s story, I would welcome even more insight on this, but even if someone asked me before reading this post, I would have been leaning towards the more opinionated, thought-heavy approach to selection instead of relying on model comparison (unless I was aiming for pure prediction).
   Point (a) seems to me what model comparison is all about, as opposed to model selection, but maybe I’ve just been lucky to learn this lesson.
   Finally, another tool for automatic inference that seems trendy among us ecologists is structural equation modeling, or at least the kind that results in plug-and-play causal diagrams based on conditional independencies. I thought it too good to be true, and I’m currently checking some posts here that seem to indicate that’s the case.

   Reply ↓

   • DAGs can be criticised of course (David Freedman complained about the “where are all the arrows” problem at least once, i.e. toy DAGs often seemingly under-connected), but they still seem like the clearest route to attempting causal inference, especially if sensitivity analyses are applied. There are of course papers using thoughtful DAGs in the ecological literature (and Grace and Irvine reviewed the area recently), but I agree that there is also a subsection of the literature that applies automatic inference to SEMs using P-value cutoffs. Who knows what type of inference that is supposed to be reaching for.

     Reply ↓

     • Thank you Anon. I guess this is the review you’re referring to https://doi.org/10.1002/ecy.2962 , I will definitely check it out, and I’m linking it here for anyone interested. Anyway yes, the kind of SEM implementation I’ve seen is with p-value cutoffs to reduce the number of dependencies.

       Reply ↓

       • Yes, that one. Myself and colleagues have also published in this area. For example: https://doi.org/10.1111/2041-210X.14492 and https://doi.org/10.1111/ddi.13698

5. Part of the issue is that biology education in general lacks any exposure to quantitative model building and assessment (except in a handful of programs). Most of the topics in Wingreen and Botstein’s excellent “Back to the future: education for systems-level biologists” article, from 2006, are still completely foreign to students. Knowing the wonders of the Luria-Delbruck experiment, for example, should stop anyone from throwing out outliers simply because they’re outliers!

   Ecology as a subset of biology is especially odd — it has lots of “modeling” and equations, but appallingly low standards for what counts for validating a model. It’s depressing to see paper after paper in ecology have an elaborate model, a cloud of featureless points, and breathless statements about some complex ecological concept being validated.

   Reply ↓

   • Spot on. The main problem in my view is that most practitioners apparently never have a clear idea of what type of inference they are groping for, hence the confusion around explaining v. predicting v. describing, and decision theoretics atop that (which is not to say there is no overlap between categories, but if you have no clear framework to build on in the first place, things are not likely to go well).

     Reply ↓

     • Agreed, especially on the ‘breathless statements about some complex ecological concept being validated,’ and that better training and higher quantitative expectations would help a lot. For now, I am going to try to start teaching ‘explaining v. predicting’ so any recommendations there would be appreciated.

       Reply ↓

       • Galit Shmueli’s paper “To explain or to predict?” is a useful light overview. I love Jim Hodges’ 1996 paper “statistical practice as argumentation” too. Some other references in this review: https://arxiv.org/abs/2308.11458

6. Thanks for the interesting writeup. From the Julia forums, I was under the impression that ecologists fit quantitative models motivated by biological assumptions that are represented by ODEs, but I guess that is a sampling bias :-P

   Reply ↓

   • Yeah, I too had the impression that ODE and Agent based models were much more popular in Ecology than in most other fields of science… Not necessarily exclusively from my Julia forum experience but also from the kinds of stuff you see in Agent based modeling books.

     Reply ↓

     • Ecologists work at different scales. You can build mechanistic models using ODEs for relatively simple systems where the agents – usually species, and not too many of them – interact at roughly the same point in space and/or time.

       The kinds of models discussed in the post are usually at larger scales: how species distributions, abundances, and diversity change over broad – think continental or hemispheric – environmental gradients.

       The challenge is knowing what variables to throw in there as “environment.” It has been a so far insurmountable challenge to use inferences from those local mechanistic models to come to a clear idea of how best to model the large scale.

       Reply ↓

       • Good point, though I think we’d get further on large-scale predictors with a better workflow and less `try everything!’ I am also not sure how far our ODEs are getting us. One of our biggest ones is Lotka-Volterra and that fits a lot of different stuff (see https://statmodeling.stat.columbia.edu/2025/03/25/lets-analyze-how-we-analyze/ and also Lakeland comment).

7. I enjoyed reading this. I find that in medicine there is still the frequently encountered and misguided belief that you can simply throw all your data into a blender and get a model that is superficially valid – ML (or AI if you’re in the C-suite or applying for funding) has expedited this process a lot, but it’s fundamentally no different than stepwise selection to minimise some function. You would think that medicine, with such a strong preference for proven efficacy, would be immune, but doctors are often as or more gullible than administration.
   Optimistically, I think the obvious success in out-of-sample prediction from thinking deeply about the functional form of the model will win out. It’s already been demonstrated empirically that a well-specified regression model can do better than many neural nets, so that may be the requisite edge to justify more widespread adoption.

   Reply ↓

8. Not sure this is the exact field you’re referring to but a very similar issue exists for environmental remediation work. It involves making many decisions based off (usually smallish) datasets to determine the state of a polluted location and what to do next (remediation, control, release, etc). I can’t speak for other places, but in the US the “automation” has been a constant dream. The EPA funded the creation of a software called ProUCL. I spits out MANY (perhaps over 20 or 30, I haven’t counted) different types of 95% UCLs. It then picks one to recommend to you based off a few GOF tests and other simple summary statistics like number of censored/uncensored results and standard deviation. From my time in the field, I see this software being required almost everywhere. Where it isn’t required, people are often still dependent on it. Millions, maybe billions of dollars, in cleanup projects in the US are guided using the output from ProUCL as it is seen by many as the “automated” solution (many projects lack even a single statistician in the process, either on the contractor side or the regulatory side).

   Reply ↓

   • To the extent that this comment is along the lines of Lizzie’s post and some of the other comments, I have the following reaction. It is not surprising that the world looks for shortcuts, automated procedures, or other ways to facilitate decision making. If the alternative is always to have eminently capable researchers do their best work, then I question the feasibility of asking for all decisions (environmental remediation in this case) be based on that undoubtedly superior process. The number of decisions surely exceeds the number of qualified statisticians to do the analysis we might all like.

     This is not an excuse for poorly constructed or executed analysis. But complaining about automated procedures seems like too easy a target to me. Frankly, I am happy to see many of the analysis methodologies become automated – it saves an enormous amount of time to not have to construct every regression model, neural network, or boosted algorithm manually – especially when they often give very similar results. I think the real effort needs to be more focused on data quality – measurement and meaningful connections between decision problems and the data available. I don’t see those parts of the process being automated easily, nor do I think they should be. I want humans to be accountable for the many judgements that are required in those steps. Some of the statements here strike me as denigrated all automation, rather than a more granular focus on where automation can be useful and where it is dangerous. At least that’s the reaction I have.

     Reply ↓

     • > Some of the statements here strike me as denigrated all automation, rather than a more granular focus on where automation can be useful and where it is dangerous. At least that’s the reaction I have.

       I think your reaction may help explain why you are having trouble seeing the point that Lizzie (and Miles, in the comments above) are making. I do not have your reaction, and I do not see the post or comments as denigrating all automation. What I see instead is a call for the appreciation that different kinds of modeling serve different kinds of goals.

       As Miles points out, a machine learning model’s only goal is out-of-sample prediction from the same population as the training data. Such a model is not well-suited to formulating or testing causal hypotheses about specific mechanisms. That does not mean, however, that such a model cannot be used to inform a mechanistic model, for example, by helping a researcher identify what components they should consider when building such a model.

       Without appreciating that different kinds of modelling serve different kinds of goals, a researcher or practitioner can be easily misled. For example, one might assume that ML-based feature selection identifies true casual relationships or that the only way to use a mechanistic model is to test it on arbitrarily held-out subsets of data. Those modeling practices are useful in specific contexts for addressing specific questions and the onus is on the modeller to make clear their goals and choose methods that serve those goals. So in that sense, I suppose Lizzie and others are arguing against automation, but against automation of critical reasoning, not against “automation” in some broader sense.

       Reply ↓

       • In that case, I am in agreement. Some of the extreme statements (“not well-suited to formulating or testing causal hypotheses,” “the only way to use a mechanistic model is to test it on arbitrarily held-out subsets of data”) confuse things a bit. I read the original post and some comments to suggest that machine learning was not useful for testing causal hypotheses compared with a more intuitive selection of potentially causative features. Or that testing mechanistic models on held-out subsets of data was useless for inferential analyses. I think those statements would be too extreme. Critical reasoning is what I want too: but the automation of machine learning models does not impair critical reasoning in my mind (this may be an area that others would dispute) – I think it can enhance and clarify where critical reasoning is most important. Automation does allow people to avoid critical thinking, but “pick your few best-guess damn variables — the ones you can make predictions about and describe the functional relationship of them to your response variable(s) and you put those in your model” may be just as damaging to critical thinking. I view automated machine learning as complementary to thinking critically about measurement and modeling choices, not as undermining these. No doubt there are people that will avoid critical thought and just use automated processes (as seems to be suggested), but I see that as something they would also do when picking their few best-guess damn variables.

9. Excellent post, Lizzie. Well said! Good thinking is hard work and has lots of exposure (in the mountaineering sense of the word https://en.wikipedia.org/wiki/Exposure_(heights) ). Those two qualities make it less desirable than automated solutions.

   Reply ↓

10. I also found this to be an excellent post.

    I guess I kind of sympathize with those seeking automatic inference. Extracting climate signal from tree rings has been a big deal for 30+ years. Despite all that work, very little progress has been made. Ecologist Jim Bouldin gives an excellent explanation of that in Part 15 of his amazing blog post series “Severe Problems in Dendrochronology:

    https://ecologicallyoriented.wordpress.com/2018/08/06/severe-analytical-problems-in-dendroclimatology-part-fifteen/

    There are many complex relationships between the variables, all somehow reduced to a single measurement, tree ring width (that’s not quite true, maximum latewood density is another variable, but it is more of a sanity check than an independent measure). And there is also contingency, a late frost or a single hot day in summer can damage the foliage and slow growth.

    When Lizzie writes “you pick your few best-guess damn variables — the ones you can make predictions about and describe the functional relationship of them to your response variable(s) and you put those in your model”…that is exactly what scientists have been trying to do for 30 years. It just doesn’t work.

    Raw tree ring width data looks terrible! We are absolutely reliant on statistical methods to extract anything.

    I will finish with a sports metaphor. The ground game has only produced a cloud of dust. Time for a Hail Mary. Let’s try automatic inference!

    Reply ↓

    • Matt:

      Agreed that reconstructing climate from tree rings is difficult. My colleagues and I wrote a paper about the problem a few years ago.

      Reply ↓

      • Andrew,
        I must have stepped back from this stuff before 2016, because I never saw this paper. Thanks for the link.

        The graphs in that paper are ugly! That is to say, the graphing and display choices are fine, it’s the data that is hideously ugly.

        The paper discusses various problems that result in variance loss. One aspect of this that is striking to me is that increasing “n” not only does not improve the results but is actually detrimental due to variance loss. If we think of an imaginary ranked list of tree ring width series, with each entry representing one tree, ranked by fidelity to the variable of interest (in this case regional summer temperature), the list might start with a data set of almost pure signal, and finish with a set of almost pure noise. (I should add for the skeptics that some individual trees show astounding fidelity to local summer temperatures.) We have no way of putting this list together, of course, but if we did, there would be an entry above which all the data is useful, and below which all the data should be discarded. We have no way of finding which entry on the list should be the last used, and it is this problem that is intractable. We cannot even show that adding a second tree ring width series to our best one improves the results. In fact, any way we try to add power to our analysis, such as model averaging, using multiple proxies, or increasing “n” just end up costing us more variance.

        Reply ↓

    • I don’t work in ecology, so I can’t comment on your dendrology example specifically, but the idea that – principled model building hasn’t worked so let’s try the luck (a Hail Mary is in large part luck) of automatic inference – is any kind of a solution, strikes me as a bit absurd. If so much ‘noise’ or lack of information precludes any kind of principled modeling, it would seem that perhaps it should be considered whether or not it’s an alchemist’s quest in the first place, no?

      Reply ↓

      • +1 on the alchemy (referencing this? https://www.youtube.com/watch?v=Qi1Yry33TQE) though I am now working on tree rings …. (so quite excited about the part 15 blogpost).

        Reply ↓

        • Interesting talk. I had not seen that. thanks

11. Speaking as an ecologist: great post! Just spot on. A pessimistic question: the problem the post points out has existed since at least the 1970s (I’m thinking for instance of constrained randomization of species x site matrices as a ‘crank the handle’ method to [purportedly] detect interspecific competition). And the problem’s been pointed out repeatedly since the 1970s. Hence my pessimistic question: why doesn’t anything change? Why does this problem keep cropping up in slightly different guises in ecology, over and over again?

    Reply ↓

12. 100% agree. One thing that I’ve been enjoying lately is the explosion of papers on causal inference in ecology. A few that might be of interest and worth a discussion:

    Siegel and Dee 2025 – Foundations and Future Directions for Causal Inference in Ecological Research: https://onlinelibrary.wiley.com/doi/ftr/10.1111/ele.70053

    Franks et al. 2025 – Ecology Needs a Causal Overhaul: https://onlinelibrary.wiley.com/doi/10.1111/brv.70029 (has a great bit on why DAGs and causal thinking are useful even for prediction that works well with your post)

    Dudney et al. 2025 (in revision) – A causal inference framework for climate change attribution in ecology https://www.authorea.com/users/867998/articles/1248554-a-causal-inference-framework-for-climate-change-attribution-in-ecology

    Arif and MacNeil 2022 – Applying the structural causal model framework for observational causal inference in ecology https://esajournals.onlinelibrary.wiley.com/doi/abs/10.1002/ecm.1554

    Airf and MacNeil 2022 – Predictive models aren’t for causal inference https://onlinelibrary.wiley.com/doi/full/10.1111/ele.14033

    And more – Butsic, Larsen, and lots of others. It’s an interesting time in the field! But it all points AWAY from automatic inference approaches like what you’re talking about, and that is 100% key.

    Reply ↓

    • Useful list. Here are two recent actual applications: https://doi.org/10.1111/2041-210X.14492 and https://doi.org/10.1111/ddi.13698

      Reply ↓

    • It’s great that so many folk are starting to understand this in ecology (although of course John Grace has been saying this for 20+ years), but whenever I see these types of high-level ecology methods review papers part of me just wonders why quantitative researchers are not already reading outside of their discipline when it comes to fundamental issues of reasoning and logic?

      Reply ↓

      • *James Grace!

        Reply ↓

        • I saw a talk by James Grace almost 20 years ago and really enjoyed it. I still remember him explaining, ‘if the grass is wet and it’s raining we cannot learn about the state of the hose, but if the grass is wet and it’s not raining, then ….’ After I read his book and did some SEMs.

          Back then we called them SEMs because everyone had figured out some rampant issues with all the path analyses ecologists were doing and so we waited 5 years and then did the same thing but called it SEM instead. SEMs seem big again, but with some extra bells and whistles and a ton of ‘causal inference’ splashed around (and often it’s not even related jargon, it’s just the phrase ”causal inference’ written repeatedly). I am not convinced any of this is helping ecology. Instead graduate students try to correct me when I refer to experiments as a way to do causal inference because they think it only applies to observational data combined with certain statistical approaches. I feel the car driving backwards ….

          Stats are great, but ecologists can often do experiments. And better than that they could develop models specific to what they’re doing that guide the experiments. Wouldn’t that be a brave new world.

        • Hi Prof. Wolkovich,

          I’m interested by your last point: “… develop models specific to what they’re doing that guide the experiments.” Could you elaborate or point to an example?

          Thank you!

13. “*We’re somehow also now calling most of machine learning ‘AI’ in ecology. Are other fields doing this?”

    I’ve even seen people refer to an elastic net linear regression as ‘AI’, which to me comes across as rather insulting to the really intelligent researchers who put together that model.

    I’ve always viewed AI as a subfield of machine learning, and machine learning as a subfield of statistics. But working with computer scientists I’ve come to realise that many of them view machine learning as a subfield of AI and AI as a subfield of computer science, where statistics is that annoying bit of maths you need alongside.

    There is no clear cut definition of what AI is.

    Reply ↓

14. Am I misinterpreting or is the ecology community (at least its machine learning maximalist subcommunity) just not that worried about inference-after-selection biases? Is care taken with regard to data leakage between the model selection stage and the fitting of the final model that will be used for inference? In this regard, your “best-damn-guess” approach to model selection seems much more prudent!

    Reply ↓

    • I fear many ecologists have never heard of inference-after-selection bias. I am not even sure how many have learned the term bias in a statistical sense. And to make myself part of the problem, as I am, I don’t teach the term ‘inference’ in my class, but I think I should start.

      Reply ↓

15. This reply is for Miles Moore (I guess there is a limit to replies on threads?)

    > I’m interested by your last point: “… develop models specific to what they’re doing that guide the experiments.” Could you elaborate or point to an example?

    I can tell you about the one I am working on this week. We’re interested in understanding how plants reach different phenological stages (budburst, ripe fruit etc.). Our best model is that they have a non-linear temperature response f(x) (something like a beta, with a minimum, max and optimum temperature) and they accumulate energy based on this curve to eventually reach the next stage (say, veraison in winegrapes as that’s the one I am working on now). We often skip modeling this process and just say that an event (reaching some phenological stage) happens in response to some temperature metric (and maybe photoperiod) and do a linear regression. That does not teach us much, but something depending on the question.

    We can fit the current model we have and have done it for decades, but often it’s hard to fit so people fix the min and max temperature and only estimate the optimum and threshold. Now that we are trying to fit the whole model at once it is painfully obvious that we have little inference on the optimum as we rarely observe temperature beyond it (and ditto but worse for the maximum) and we never approach the minimum for veraison. This tells me we critically need experiments at higher temperatures and maybe lower temperatures (though those are really hard, so we may also want to evaluate what is happening at those low temperatures that we’re modeling).

    Reply ↓