Keith O’Rourke’s final published paper: “Statistics as a social activity: Attitudes toward amalgamating evidence”

Posted on by

Keith O’Rourke passed away two years ago. Here’s his obituary, which was sent to me by Bart Harvey:

Lover of earth, wood, water and fire, Keith left us after a brief illness on November 27, 2022. Born to Evelyn and Frank O’Rourke, he was the second of four sons. He met Marlene and they shared their life together for 39 years.

Keith worked in landscaping throughout his undergraduate years at the University of Toronto, then at Moore Business Forms, and in the western and northern provinces and territories in the field of compressed air before returning to UofT to complete and MBA and undertake an MSc. For many years he worked as a biostatistician at the Toronto General Hospital and the Ottawa Hospital on numerous studies in the fields of cancer, diabetes, SARS and infectious diseases research before completing a PhD at Oxford University in 2004. Having worked at Duke University, McGill and Queen’s, he joined Health Canada in the late 2000’s as a biostatistician, initially in health care and then in pesticide management. A conscientious intellectual and deep-thinker, Keith endeavoured to make our world a better and safer place.

Known to enjoy the occasional three fingers of Glenfiddich or an IPA, Keith pondered the mysteries of health and safety while taking longs walks in nature, nurturing many maple, birch and oak saplings, cutting up downed trees at our cottage in the Lanark Highlands, building bonfires and stoking every wood stove he had access to. A black belt in Kung Fu, and an assistant coach in boxing at Oxford he was known to jump in the ring and spar with up-and-coming kick-boxers into his mid-sixties, until Covid ended access to the ring.

We had an unpublished project together which we had not touched since 2016. I recently revised the paper, and it will be published. The article was originally titled, “Attitudes toward amalgamating evidence in statistics”; the final version is called, “Statistics as a social activity: Attitudes toward amalgamating evidence,” and here’s the abstract:

Amalgamation of evidence in statistics is done in several ways. Within a study, multiple observations are combined by averaging or as factors in a likelihood or prediction algorithm. In multilevel modeling or Bayesian analysis, population or prior information are combined with data using the weighted averaging derived from probability modeling. In a scientific research project, inferences from data analysis are interpreted in light of mechanistic models and substantive theories. Within a scholarly or applied research community, data and conclusions from separate laboratories are amalgamated through a series of steps including peer review, meta-analysis, review articles, and replication studies.

These issues have been discussed for many years in the philosophy of science and statistics, gaining attention in recent decades first with the renewed popularity of Bayesian inference and then with concerns about the replication crisis in science. In this article, we review amalgamation of statistical evidence from different perspectives, connecting the foundations of statistics to the social processes of validation, criticism, and consensus building.

I’m pretty sure that this will be Keith’s final published article. I very much regretted not having him around when revising the paper; we would’ve had lots to talk about.

21 thoughts on “Keith O’Rourke’s final published paper: “Statistics as a social activity: Attitudes toward amalgamating evidence””

  1. I didn’t know Keith died. His LinkedIn profile is up. I spoke with him when I was selling automated predictive modeling software. He was first rate. Most statisticians were hostile to and threatened by any automation, for obvious reasons. Keith was thoughtful. He provided suggestions as to aspects of the entire modeling process where automation could be helpful. I will read this article. It looks like it deserves a wide audience.

    Reply ↓

    • Joey –

      I’m pretty much always on my Android phone at this site. I have often found that PDFs get downloaded even when there wasn’t necessarily an indication they had been downloaded. Maybe check the downloads folder in your browser.

      Reply ↓

  5. This article turned out to be a toughie. You guys lost me a bit when you were stringing together the terms and scenarios. Active scientific realism is a new one in this context. I blame my lack of background knowledge. I worked in industry where most the client spend was on solutions taking the algorithmic pure prediction approach.

    In the article, the elephant in the room from a social science perspective is data quality and suitability. I agree that “one must weight and adjust data in light of what is known about the quality and representative of the measurement and in light of the consistency of different data sources with available research hypotheses”. You also bring up the moral hazard involved when all modeling decisions are up for grabs. I am not so philosophically minded but I think that abuse of these two concerns are so pervasive in sociology (the discipline I focus on) that the field uniformly fails the data corresponding to reality test.

    I don’t know this literature well and what I write is based on the ‘smell test’ and without much conviction. Didn’t Otis Dudley Duncan and Peter Wench have important things to say on this topic?

    I am confident that a lot of the data sociology historically used is way off.(E.g. the Wisconsin longitudinal study is really just the social history of white northern European men in Wisconsin). The data used by the Harvard sociology department impact of work “Shift Project” is emphatically inappropriate. I know because I work in the industry. The Princeton fragile families (or whatever it is rebranded to) project is problematic. They should read this article and consult with experts like you and Professor Sobel about how to do apply evidence amalgamation.

    Finally, there is little ideological division in sociology. The spectrum runs from Bill Ayers to Rick Perlstein to Barack Obama. There is the methodological divide and I am firmly in the camp of the strong anti-positivists.

    Reply ↓

    • Hi Joey,

      I am a Sociologist my self and I think we agree on many things (although I don’t understand everything you write), now after reading your post I was really surprised by this: “There is the methodological divide and I am firmly in the camp of the strong anti-positivists.”. I am in the positivist camp and I am surprised that an anti-positivist even reads a statistics blog.

      So for me science is about finding theories: A theory is a set of statements that are logically connected and that makes interesting predictions. A theory is better if it is more general (applies to more cases) and leads to better predictions. A simpler theory is also better (what ever this means exactly). This is, for me, the core of positivist thinking. For me this in closely aligned to quantitative methods.

      For me, anti-positivists do not even try to find general theories, they are more concerned with the particular and they don’t really have standards what counts as a theory etc.

      What do you think about this? Maybe you have a very different understanding of positivism? I must admit, I am not sure what it means and quickly looking at wikipedia https://en.wikipedia.org/wiki/Constructivism_(philosophy_of_science) suggests that my understanding may be wrong.

      Reply ↓

        • Hi Joey, I read your article. It is a bit weird because there are no sources and no arguments as far as I can tell. You are merely asserting that causal analysis methods don’t work and we can only describe social phenomenon, rather than explaining them. I agree that current methods are a joke and don’t work (if I hear again someone presenting his “fixed effects panel regression” at a conference I’m gonna throw one of the many paper aeroplanes I made out of boredom).
          The social world is, however, governed by causes, much as the physical world. Of course predicting is harder, because many things can influence social phenomenons and these mechanisms vary across periods and places. Still, causes are there and often we will have some idea of what they may be and how they could influence the outcome of our study. This we should incorporate into our models. Besides, if we use quantitative methods only to describe, we still have to come up with theories. But how do we do that?

          Anyways, I think these abstract discussions are not so fruitful and it is better to look at concrete examples. Which phenomenon can be studied by mathematical models and which can’t? For example it may be impossible to predict the next major terrorist attack, but it is possible to predict the next economic crisis. The financial crisis of 2007 – 09 for example was predicted (although only by a handful).
          See here: https://en.wikipedia.org/wiki/2007%E2%80%932008_financial_crisis#Prediction_by_economists

  6. Over the years, Keith wrote to me several times as a result of some comments I posted on this blog, and gave me good advice on evidence synthesis. He also had me read the Evans 2016 book, which I found hard going. He also sent me his own papers to read, which were also very helpful. I greatly appreciated his willingness to educate me on evidence synthesis.

    Some comments on the paper itself. I think the paper is hard to read because there is a lot of unspoken context behind it, which one can only understand if one has followed Andrew’s work over the years.

    Some specific points:

    > targeting a uniform(0,1) distribution of p-values for all nulls

    It would be helpful to spell this out. Targeting a uniform(0,1) distribution seems like an underinformative expression.

    An actionable suggestion in the paper could have been placed more prominently (e.g., in the abstract):

    > A cleaner approach would be to analyze larger data sets directly, not by postprocessing published
    estimates and p-values but by modeling larger and more diverse sets of raw data.

    > the idea that variation can itself be quantified
    is central to any statistical understanding of modern social and biological sciences.

    And then what? It would have been good to talk about what one can do after one has succeeded in quantifying variation. In my view, once one starts to understand the extent of the variability, one becomes increasingly skeptical that there is a single common reality that can be found. I face this problem when reviewers ask about my papers, did you find out something true about the world in this study? The truth is a moving target; and only accumulation of evidence will give some reliable sense of what the central tendency is (along with the uncertainty of that estimate). I think that this is also one of the points of this paper.

    Discussion of Simpson’s paradox: the figure should have been shown here. It’s hard to follow the dscussion without knowing what the figure looks like (what if one is reading the paper on a German train with no internet? It happens).

    I was a bit surprised to see that this paper will appear in an MDPI journal (Entropy); or am I mistaken about this? I’m really surprised that Andrew decided to publish with this publisher. He has published another paper with Michael Betancourt in Entropy, which also really surprised me. I would never publish with MDPI, for reasons that can be googled.

    Reply ↓

    • Shravan:

      In answer to your last paragraph: In 2017 I was contacted by Entropy and asked to write an article for a special issue. The editor of the special issue was someone I’d corresponded with before on a statistical topic, and the whole thing seemed legit. The journal seemed a bit “cranky” in representing some small corner of the research world, but not “spammy” in the sense of being a fraud. At the time I’d been talking with Dan Simpson and Mike Betancourt about various Bayesian topics, and this seemed like a convenient opportunity to write something that might spread our pragmatic Bayesian ideas to the niche audience of the people who work on maximum entropy, so we quickly wrote something up. I like the resulting paper, The prior can often only be understood in the context of the likelihood, and I suspect it would’ve been difficult to place in a traditional journal. I guess it had a good title, because it’s been cited a lot since then.

      Then in 2023 my colleague Yu-Sung Su informed me that he had sent our article, Who wants school vouchers in America? A comprehensive study using multilevel regression and poststratification, to Social Sciences, another MDPI journal. This was a paper we’d written around 2010 that I kinda liked but was never otherwise gonna be published, so I said sure, no problem. I doubt it’s ever going to get much notice—I guess that posting on Arxiv would’ve been enough—but, whatever.

      I sometimes get emails from Entropy or other MDPI journals asking for me to review or submit a paper or edit a special issue or whatever, and I always ignore them. But then in 2024 I received a request to submit a paper to a special issue of Entropy, as a personal favor to someone, so I said, Sure, and I looked for some already-written paper I could send to them that wouldn’t otherwise be published. First I sent them one thing which had been kicking around for awhile but then one of my coauthors didn’t want to publish in an MDPI journal—fair enough, and this motivated then to revise and update the paper and send it somewhere else—and then I remembered that paper with Keith, which seemed perfect. So I cleaned it up a bit and sent it off. But, yeah, I wouldn’t have sent the paper there had it not been for that particular request.

      Reply ↓

      • I think these MDPI people are exploiting you and your name to gain credibility. You can easily post the same papers in arXiv or in a more reputable publisher’s journal. It’s not like you need publications at this stage; and people will read and take seriously arXiv papers as much as any published paper, if the paper is good and useful.

        Reply ↓

        • This morning, what’s in my inbox? Oh, like clockwork, it’s an email from Entropy HQ in Wuhan “in recognition of my well-respected work within the field”. I’m not sure what field this means (they don’t elaborate), unless it is the field of having commented on Prof Gelman’s blog. My view is that if it looks and sounds like a scam, it either is one, or is not going anywhere any time soon, as so many people will distrust it. Kudos to Andrew for giving them a chance but I tend to agree with Shravan on this, we have better things to do than dealing with hustlers.

  8. I’ve only read the abstract and skimmed the paper. I was a bit surprised to see no mention of Brad Efron’s 2010 *Statistical Science* paper, “The Future of Indirect Evidence” (volume TOC: https://projecteuclid.org/journals/statistical-science/volume-25/issue-2). I found the paper, combined with the discussion, interesting and illuminating. In particular, I like Efron’s term “indirect evidence” for how information is shared in the context of hierarchical (and empirical) Bayesian modeling. I was especially surprised to see no mention of the paper or the phrase because Andrew wrote a very good comment as part of the discussion! Maybe it’s because of the paper’s emphasis on the social processes related to amalgamation of evidence. Multilevel/hierarchical modeling appears to play a more important role in the abstract than in the actual paper.

    Another favorite phrase in this context is due to Tukey. “Borrowing strength” is fairly well known, but when you look into his writing, the actual phrase he used for indirect sharing of evidence across related cases was a bit more evocative: “mustering and borrowing strength.” I like the image of “mustering” the sources of evidence together (and it ties in nicely to “aggregating”).

    Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *