Category Archives: Causal Inference

JAMA rules of engagement on causal and predictive language

Posted on by
27

Erik, Witold, and I recently published a paper in the Journal of the American Medical Association. This was cool. A day after the paper came out, I ran into someone I know on the street and he said he liked our article. That’s kind of amazing, as usually it seems that nobody reads anything. We’ll have to publish in JAMA more often.

As part of the publication process, we were sent some general reporting requirements per JAMA standards. Here’s the part that will interest you:

USE OF CAUSAL LANGUAGE
Use of causal language (including use of terms such as “effect,” “efficacy,” “cause,” and “x increased y”) should be used only for randomized trials. For all other study designs (including meta-analyses of randomized trials), methods and results should be described in terms of association or correlation (eg, “x associated with an increase in y”) and should avoid cause-and-effect wording.

Avoid use of the words “predict” and “predictive of” for reports of observational studies assessing independent and dependent variables, unless your study design is a diagnostic/prognostic study and has used an appropriate approach to quantifying predictive performance such as (but not limited to) split sampling, cross-validation, or bootstrapping.

On the first item . . . Wow. I wonder if they’d let you write, “Cigarette smoking causes cancer in humans”?

I guess the second item is ok if it’s interpreted broadly enough. For example, Bayesian methods use an appropriate approach to quantifying predictive performance, so I guess that would be included in the “not limited to” category.

Getting better causal inferences by using more information (my talk at Stanford this coming Wed)

Posted on by
4

It’s the political economy seminar at Stanford business school, Wed 14 Oct at noon in room M109:

Lagged regression, difference in differences, two way fixed effects, regression discontinuity, placebo controls: these are different methods used when estimating causal effects from panel data. We’re working on a general approach using multilevel regression and poststratification (MRP) that aims to get the best from all these approaches. We demonstrate with a range of examples, including observational studies in economics, political forecasting, and controlled experiments in biology. This is joint work with Jori Korpershoek and others.

Come and ask me some tough questions!

The talk will be pretty open-ended. We have lots of ideas and some successes (for example, the chickens paper) but it’s mostly open questions and promising research directions. Should be fun.

I won’t be talking about this paper or this paper at all—that’ll have to wait until I’m invited to speak in the English department.

Hey—we’re hiring two postdocs!

Posted on by
3

I’ll post more formal ads soon, but, very briefly:

One project is on generalization for causal inference with a focus on education research, following up on MRPW: Regression, poststratification, and small-area estimation with sampling weights.

The other project is on Bayesian inference in scientific workflow with a focus on contamination models, following up on this work on design and analysis of bioassays.

Both projects are application-focused—that’s how we do statistics—but the importance of this work is not at all limited to these particular applications. This research will have relevance in policy analysis, medicine, and scientific measurement and inference more generally/

Further details soon.

When are Causal Inference Methods Needed to Answer Causal Questions?

Posted on by
2

Earlier this year we discussed Donna Spiegelman’s talk, “Rethinking SUTVA and Causal Identification: An Epidemiologist’s Perspective,” which she summarized as follows:

She’s giving a followup talk online on Mon 12 Oct, 3:30pm. Here’s the abstract:

When do specialized causal inference methods add value beyond standard approaches? Drawing from her unique perspective as both an epidemiologist and a biostatistician, Donna Spiegelman will consider the circumstances under which commonly invoked causal assumptions are necessary for causal inferences to be validly made from data, concluding that often not. She will show that valid learning can occur 1) under conditions much less restrictive than required by current widely used methods, 2) when real world implementation of interventions vary, and 3) when interventions spill over to others not directly exposed, thereby obviating components of the SUTVA assumption. She will provide evidence that measurement error is the major source of bias in observational research, not confounding, whose bias is rather tightly bounded. Finally, she will discuss the eternal challenge in science: after exhaustive efforts to collect data to predict important outcomes, a substantial proportion of the variation in occurrences of these outcomes appear to be entirely random.

The talk will be followed by a discussion. Click on the link to register (or show up in person if you’re at Yale).

Here’s a causal inference problem: where did the estimate of $64 million come from in “Blake Lively Says She Suffered $64 Million Financial Fallout From Justin Baldoni Conflict . . .”?

Posted on by
6

OK, here’s a causal inference problem for ya:

Blake Lively Says She Suffered $64 Million Financial Fallout From Justin Baldoni Conflict . . .

In Lively’s memorandum in opposition to Wayfarer Studios’ motion in limine filed April 17, her expert Dr. Ashlee Humphreys–a professor of marketing communications at Northwestern University–alleged Baldoni’s use of “the retaliatory phrases ‘tone deaf,’ ‘bully,’ and ‘mean girl'” during the alleged smear campaign against her had amounted to “176,738,781 impressions, in an amount of approximately $36.5 to $40.5 million.”

The filing also alleged that “retaliatory statements” made by the It Ends With Us director’s attorney Bryan Freedman during the legal battle had amounted to “116,959,530 impressions, in an amount calculated at $24,375,267.”

They continue:

Lively’s expert Humphreys had provided the calculation for reputational harm damages by quantifying the reach of the alleged phrases and statements, identifying “the number of impressions providing the basis for each” and calculating “particular dollar damages.”

I’m curious how the expert computed these dollar effects. I did some googling but couldn’t find the report with the numbers, so all I can do is speculate. Let’s do some divisions:

> c(36.5, 40.5)/176.738781
[1] 0.2065195 0.2291517
> 24.375267/116.959530
[1] 0.2084077

So, something like 20 cents per impression? I wonder where that came from. Maybe they did some sort of regression analysis estimating the effect of negative impressions on celebrities’ Q scores, and then attributed that to Lively not getting some roles that could have been worth $40 million and $25 million? The only thing is, the fact that both ratios are close to the same number suggests that they’re doing a straight-up calculation assigning a dollar cost per impression, but that can’t be right, as ultimately it has to result in foregone income, no? Also I don’t get why one of the estimates is a single number calculated to the nearest dollar value and the other is a range of round numbers. Also, once you have a range, I’m surprised it’s so narrow. Where does “approximately $36.5 to $40.5 million” come from? Why not “$25 to $50 million”?

If anyone has a copy of this expert report and would like to leak it to me . . . you have my email!

No joke–I’d like to see the analysis they did to justify their numbers. I’m not saying their estimated damages are wrong; Blake Lively is a big movie star, and her lifetime earnings could be huge. I’m just interested in the methodology here.

P.S. We’ve been assured by an expert from the National Academy of Sciences that scientific citations are worth $100,000 each. So now we can convert that to social media impressions (500,000 impressions = 1 citations) as well as more traditional cost benchmarks such as Jamaican beef patties, publications in Nature Communications, naming rights for a center for junk science, and, of course, attendance at a conference featuring Grover Norquist, Gray Davis, and a rabbi.

“Phonics is a fad, but it’s also good, so we should do it anyway, even though it will probably disappoint people”

Posted on by
32

Tyler Watts and Drew Bailey write:

Educational policy is often viewed as a field of fads, where one follows another for seemingly arbitrary reasons. We might like to believe that fad-based policy could be superseded by evidence-based policy, with rational actors choosing policies because of scientific evidence. But that’s been hard to achieve.

In the cynical version of the story, hucksters overpromise on their pet projects, we make the changes they advertise, and they get rich while the public is left disappointed. This probably happens more often than it should, but this narrative ignores the conditions that make such disappointment ubiquitous in educational policy.

Why? They explain:

Even if the hucksters all disappeared tomorrow, we’d still have this hype cycle, because overpromising seems inevitable when effects are small, and political disappointment will always follow. . . .

The educational policy research ecosystem usually selects for safe, often evidence-based, politically feasible, but ultimately oversold ideas, and we think that hurts our ability to make progress. In education, real effects are really small. . . .

Why are the effects so small? The outcomes we like to measure, such as reading comprehension, depend on many personal and environmental factors, making it hard for a discrete change in schooling to make much of a difference. . . .

This is a well-known problem in education research. For example, it came up in my 2018 talk at the Society for Research on Educational Effectiveness, and the point wasn’t new then either!

But, as Watts and Bailey explain,

The extreme education-nihilist position—that educational improvements can never affect students’ skills—is actually wrong. Endorsing it is likely to lead to poor education policy. The edu-nihilist position overlooks that small effects are not always zero effects. Small effects can actually be meaningful and, in some cases, worth it from a cost-benefit perspective for society.

One way to think about this is to consider classroom teaching.

I work my butt off every semester teaching my classes. Every class takes a lot of effort, both by me and the students. But any given class can’t have much of an effect on total learning. Indeed, the entire semester doesn’t have such a large average effect, when you consider that some students already could do most of it before the semester began, some still don’t get it when the semester is over, and others learn some things which they later forget. (On the plus side, students can learn all sorts of useful things that don’t happen to be on the exam.) But the effect of these classes is not zero. Students take them and they learn some things. So of course there’s no simple switch that will improve performance by 0.2 standard deviations. We’re already doing what we can. And the flip side of this is that a small improvement is not nothing. We teach each class knowing that at best it is giving a small average improvement, but it’s still worth doing.

Watts and Bailey continue:

When education policy researchers say something “works,” most people assume that effects are much larger than they are. . . . In most cases, researchers and policymakers fall prey to effect size magnification because they develop real sympathy for a pet program. When this happens, they put forward the most promising evidence, which is likely biased. The field then develops unrealistic expectations, and researchers, policymakers, and advocates downplay or withhold information about their uncertainty.

Hey, here’s an example from celebrated economist James Heckman, who doesn’t have much of an understanding of selection bias and goes around promoting wildly exaggerated estimates of early childhood programs.

Here are Watts and Bailey on what happens next:

Educational policies are uniquely poised to fall apart when overhyped, because public scrutiny of educational outcomes is so intense. Because most of the variance in student achievement comes from factors schools can’t control, overselling sets up an inevitable letdown. When disappointing results emerge and large gaps in achievement remain, we overreact to the apparent failure. Feeling the heat, the researcher, policymaker, or advocate might retreat and say that whatever was promised was never supposed to be a “panacea.” This always feels cheap because the language rarely shows up in the press releases, grant proposals, white papers, or speeches preceding the policy. Ultimately, we are left abandoning ship, even if the policy in question was actually helping.

The clearest example of this is probably the accountability reforms pushed by No Child Left Behind. Everybody hates No Child Left Behind because it was terrible and didn’t work! Right?

Actually, our best policy evaluations suggest it did work; it just had small effects, which is exactly what we should have expected. Instead of taking the small effects as a sign that something is working and figuring out ways to improve it, politicians and advocates decided it was a failure and moved on to other things.

And then they turn to the latest trend:

In a country that can’t agree on much, states from Alabama to California have adopted policies designed to promote the science of reading. . . . Count us as phonics believers. Kids need to know how to sound out words when they learn to read. We taught our own kids how to read this way, and we ourselves even learned to read this way in the pre-science-of-reading dark ages of the 1990s. But the unfortunate reality is that we have no reason to believe a sudden shift to a phonics-based curriculum will significantly affect reading outcomes in this country.

There are several reasons why.

First, meta-analyses of phonics interventions show fadeout. This means that kids who get phonics instruction for some period of time look a lot like kids who didn’t get phonics instruction in terms of reading just a year or two after the intervention ends.

Second, teaching phonics seems to help with decoding and word recognition. This is an important building block of literacy, but there isn’t much evidence that it helps with eventual reading comprehension—the ultimate goal. . . .

Finally, we are fooling ourselves if we actually believe that phonics hasn’t been used in classrooms up to this point. Do we really think that it never occurred to teachers in the “woke” states to help kids break words into syllables and sound them out? The counterfactual condition here has become a strawman. . . .

The best evidence on what happens when states adopt early literacy policies is probably this working paper, which finds effects of up to about .10 standard deviations . . . on high-stakes tests for 3rd graders (that’s good). They didn’t find much on low-stakes tests administered to 4th- or 8th-graders.

So, if we enter a halcyon period for phonics instruction, we might help a lot of kids learn basic decoding skills when they are first learning to read. But . . . effects will be smaller than we were led to believe. Importantly, they probably won’t move the needle much on outcomes like NAEP trends, which compare U.S. cohorts of 4th and 8th graders to each other. . . . The question is: how will we interpret this when the sobering news starts to come in?

Just in case we weren’t clear: doing more phonics in early-grade classrooms is, for the last time, probably a good thing.

They conclude:

Education policy is not the only field that faces conflicts between public and expert opinion. But the combination of big expectations, small effects, and constant attention to children’s educational outcomes means education policy is likely to keep experiencing these conflicts.

What can we do to mitigate this educational political economy problem? . . .

We should ask interested parties to make predictions on the record about how these children will differ (and by how much). And our society should reward researchers and policymakers whose mental models are closer to reality, rather than those who are best at selling potential solutions in the first place.

And my own thoughts

I agree with everything that Watts and Bailey wrote above, and also have a few things to add:

First, there’s my recent paper with Amy Krefman, Lauren Kennedy, and Jessica Hullman, Hypothesizing an effect size by considering individual variation, where we try to think systematically about how to get more realistic expectations of average treatment effects.

Second, a point from my above-linked SREE talk, that research should be less isolated from practice. From one direction, research studies should be more realistic, more connected to the real world of education—this is a point that Watts and Bailey make in their post. From the other direction, general practice should follow principles of research: students, parents, teachers, and administrators should be empowered to make choices and experiment, and, along with this, choices and outcomes should be measured so that effectiveness can be studied. We should all be trying to do better anyway, and we should design our data collection so we can all best learn from these experiences. As we discussed in a different context, local data collection, collaborative analysis, and local recommendations.

Third, ummm, this is a longer one so I’ll just quote myself:

You can conceptualize an education intervention as a vector, where the direction of the vector is the material being learned and the length of the vector is the amount that students are motivated to work. You want the material learned to be useful—you’d like the vector to have a positive “dot product” with the vector of skills, knowledge, and understanding that will be useful going forward—but, conditional on those two vectors being roughly aligned, the real gain is in the magnitude. And this magnitude will be an interaction between the teacher and student: there’s no button to push to create motivation, and if there were a button it would already have been pushed.

What I’m saying is that, when thinking about acupuncture, or physical therapy, or coaching, or teaching, we have to go beyond what I’ve called the penicillin model of science, the idea that innovations come from nowhere and that the job of statistics is to design and analyze experiments to reject the null hypothesis of no effect, and in which the treatment in such experiments is considered as a black box, with the goal being to estimate an average treatment effect.

I don’t think the penicillin model usually applies. Most of the time in health, education, and just about any field, improvements are incremental, and the goal is to improve the process while gaining understanding. Clinical trials and offline experiment both play a role, and you’re not going to learn much by studying a treatment as if it’s a black box.

This is not to say that there cannot be new developments in any of these fields, nor is it to deny that such developments can sometimes arise serendipitously. I just think that, in any case, you have to go beyond the average treatment effect and think about the mechanism of action.

OK, here’s a statistics problem for you, ripped directly from the headlines!

Posted on by
21

From the NYT:

Mayor Zohran Mamdani is expected to propose linking Grand Army Plaza with Prospect Park by closing a dangerous stretch of road between them. . . .

The plan would effectively reconnect the oval-shaped plaza’s most prominent feature, the Soldiers’ and Sailors’ Memorial Arch, with Prospect Park, Brooklyn’s emerald jewel, restoring the 80-foot-tall arch as a grand entrance to the park, as its designers intended.

Cars would be banned from the parts of the southern end of the plaza that border the park, from Union Street to Eastern Parkway, which would do away with a forbidding four-lane crossing, according to the city Department of Transportation. . . .

After public feedback, much of which favored more dramatic changes to the area’s roadways, the Mamdani administration selected a plan that would remove cars from the lower tip of the plaza and divert some traffic to other streets.

They quote people on both sides, who raise a number of points:

Lynda Balsama, an organizer with . . . United Neighbors of Prospect and Crown Heights, said the redesign would divert more traffic to nearby residential streets and add travel time for fire trucks and ambulances. . . .

Between 2021 and 2025, there were 219 traffic injuries along the plaza’s central roadways and outer ring, according to the Transportation Department. . . .

In spite of the constant traffic, the plaza remains a popular destination for those who want to admire the work of Frederick Law Olmsted and Calvert Vaux, the designers of the plaza and Prospect Park (as well as Central Park in Manhattan). . . .

“Every time N.Y.C. D.O.T. has provided more space to pedestrians at the park, it’s been an instant success, and it becomes impossible to think of how the space could have functioned before,” Mike Flynn, the transportation commissioner, said in a statement about the plan. . . .

The Transportation Department also expects the plaza redesign to help speed up some of the slowest buses in the borough, including the B41, which serves over 27,000 daily riders, because the reconfiguration should reduce congestion in and around the loop.

Here’s the official announcement from the city government.

Here are the issues raised above, stated more abstractly:
1. Current public opinion,
2. Worse traffic on other streets,
3. More fire damage (because the fire trucks are delayed) and more deaths or serious health problems (because the ambulances are delayed),
4. Fewer traffic injuries at the intersection in question,
5. Happier tourists and locals who want to enjoy the park,
6. Track record that past pedestrian expansions have been popular,
7. Faster buses.
I’m sure you could think of more; this is what I got from the news article. Of these seven issues, five are pro-redesign and two (#2 and #3) are anti. That doesn’t mean the plan is a good idea, as it reflects the balance among who was quoted. That said, I did some googling and it’s hard to find much opposition to the plan, so maybe the Times had to work to get even that one quote.

In any case, the research challenge is to put numbers on all these things. This still doesn’t answer the question of what to do, as it involves balancing questions of public opinion, economics, convenience, and life and death. But it would be a start. It’s hard for me to imagine that item #3 counts for much at all, but it should be possible to give some estimate of the components that go into it. Indeed, it could be that the fire trucks and ambulances would actually come faster.

Anyway, it seems like a good example, to work through the numbers on the seven items above.

Metascience corner: What can we learn from 12 million empirical research results?

Posted on by
6

He approves

This is Witold and today we are building a bear.

In our recent paper with Erik van Zwet and Andrew, A statistical case for qualified scientific optimism, we fit metascientific models that use hundreds of thousands of z-values from empirical research. We will be blogging about this paper in the coming months, but the data that we needed for this paper seemed so interesting, that they spun off into a whole separate resource, Benchmarks of Empirical Accuracy in Research. I think readers interested in metascience will find it useful.

BEAR is an open-source compilation of 26 (and counting!) documented metascience datasets. Sometimes it just re-uses data from publications, like Kevin Lang’s work on false positives in economics—and dozen more. (I can’t stress this enough, huge thanks to the legion of researchers who created/maintain these datasets.) For some others I do much more myself, like processing outcomes from tens of thousands trials from clinicaltrials.gov.

And you can access all of that, 12.5 mln results in total, with a few clicks. Just go to https://witold.xyz/BEAR/. BEAR has bespoke datasets from medicine, neuroscience, psychology, ecology, education, economics, political science and more—all standardised and ready to use. Some sets have only z-values, but some have much richer structure: study types, subcategories, effect sizes, type of measures, meta-analytic groupings, etc. Of course quality varies, but I hope the documentation/standardisation will help people pick the right dataset for their research question.

Why does this exist? As I say on the website:

Quantitative metascience drives important debates about research standards. Most of its crucial contributions have been based on analysing individual datasets, often painstakingly constructed by researchers. Thanks to their work we now have better understanding of replicability, publication bias, p-hacking, reporting practices, pre-registration, significance rates, and more. But many canonical findings of metascience are based on individual datasets, each constructed in a specific way. How generalisable are these findings?

In other words, metascience itself can suffer from one of the crucial metascientific concerns: external validity. As someone who works on evidence synthesis I am very concerned with this. To give one example, Erik blogged about the virality of the scary publication bias histogram in biomedical journals. But different metascientific sets give completely different pictures of selection:

I am NOT particularly interested in selection in this post. We could talk about any other metascientific property, e.g. p-hacking, low power. I am also not trying to explain what drives these differences or say that these sets are comparable. (We get into that a little bit in the paper and if people are interested in that particular question, I can write a follow-up post.) The point is that we can learn much, much more when we go beyond one data point. OK, that’s pretty trivial thing to say, but Erik’s example clearly shows that it’s one that hasn’t fully sunk in. So I just hope that BEAR will make it easy to compare metascientific corpora and give people a better picture of the limits of generalisability of various claims. Or generally make metascientific research easier.

For completeness, the GitHub repo also includes the model that Erik, Andrew and I use in the paper I linked earlier. You can use it yourself to e.g. estimate (idealised) expected replication rates, but the model is optional and separate from BEAR data. Big kudos to Erik who got this process started and hunted down many of these sources for the paper. But most importantly the credit goes to teams of researchers who generated and maintain the linked sets.

I also have no doubt that in putting this together I made many mistakes. Even with automated checks, it’s not easy to work with so many data sources in parallel, so I’d be grateful for any corrections!

Update on a regression discontinuity dispute: some asynchronous collaboration

Posted on by
16

Anjali Thomas writes:

I am writing to share a paper which is a re-examination of my 2018 AJPS article entitled “Targeting Ordinary Voters or Political Elites”  which was previously discussed on this blog here.

The paper, written in the spirit of Gelman (2022), conducts a thorough re-analysis of my earlier work in light of recent developments in regression discontinuity designs (RDD). It also directly addresses specific critiques of the article raised both in subsequent academic literature and in previous comments on this blog.

A full response to each critique raised on this blog appears in Section A.3 on page 48 of the paper. Among other things, the blog critiqued the use of the global fourth order polynomial, and commenters highlighted that it appeared to be picking up noise in the data rather than a true relationship. While the original article did present results in the SI showing robustness to a local-linear specification with alternative bandwidths, the currentpaper significantly extends these checks and presents new results showing:

  • Polynomial & Bandwidth Robustness: Shows stability across lower-order polynomials, alternative bandwidths, kernel choices, and a donut-hole approach.

  • Noise Reduction via Aggregation: Aggregates data to the level of the running variable to reduce noise, presenting new scatter plots where the visual discontinuity persists at this level of aggregation.

  • Spatial Structure: Demonstrates that reported balance/placebo issues in recent re-analyses stem from ignoring within-constituency clustering and predictive controls.

A link to the paper is here (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7272598) and the abstract is below:

This note reexamines Thomas (2018), which advances and tests the ‘elite cooperation logic’ whereby national politicians target resources along partisan lines to win over the cooperation of co-partisan state legislators in implementing development projects. Consistent with this logic, a close election regression discontinuity design (RDD) uses project-level data to show that national legislators in North India allocate systematically higher public works expenditures to constituencies of co-partisan state legislators in the period after a state election. Responding to critiques in subsequent literature of the RDD approach used, this note shows that both the evidence of covariate imbalance reported in Bicalho et al. (2026) and the high proportion of significant placebo estimates reported in Albada (2025) are artefacts of ignoring within-constituency clustering and omitting predictive controls. Meanwhile, this note presents re-analyses confirming that the core findings in Thomas (2018) are robust to covariate inclusion using either constituency-level clustering or constituency-level aggregation. Acknowledging the problems related to global fourth order polynomials (Gelman and Imbens, 2019; Albada, 2025), the results in Thomas (2018) are also shown to be robust to lower-order polynomials, alternative bandwidths, alternative kernel choices, the donut hole approach, and an inference procedure that adjusts for worst-case bias (Stommes et al., 2023). The note re-establishes the credibility of the substantive findings in Thomas (2018) and highlights the importance of accounting for spatial clustering in both estimation and diagnostic checks in RDDs.

Anjali took my Bayesian statistics class back in 2006! It’s great to see what former students are doing, and I love seeing this sort of asynchronous collaboration.

Regarding the regression discontinuity issues, I do not think it makes sense to fit unrelated curves on the two sides of the cutoff. So I prefer the versions that fit a single curve plus discontinuity, not two curves.

The other thing I always recommend (see this recent paper with Imbens is that regressions include not just the forcing variable but also other pre-treatment predictors. This should help with both bias and efficiency.

All the discussion of bandwidth, functional forms, etc., can be wasted if the fitted model makes no sense or if it’s a distraction from including additional pre-treatment predictors.

In any case, it’s great to see this sort of open exploration.

I recommend this book on Applied Regression and Causal Inference

Posted on by
5

This post offers two recommendations and a story.

First, the recommendations.

• Here’s the book referred to in the title of the post. I highly recommend it! My only regret is that we didn’t call it Applied Regression and Causal Inference, cos that’s what it’s about.

• I also recommend our book, Active Statistics, which has literally hundreds of stories, class-participation activities, computer demonstrations, and discussion problems. A great teaching resource and also just a good read. Really!

Now, the story.

This came in the email one day:

Dear Professor Andrew Gelman,

I hope this message finds you well. I’m part of the Cambridge University Press sales support team and I work with your local Cambridge Sales Representative ** **@cambridge.org

We wanted to follow up regarding your course(s) for Fall 2025 and ensure you’ve had a chance to review Regression and Other Stories, 1 ed by Andrew Gelman/Jennifer Hill/Aki Vehtari as a potential option for your course materials.

If you’ve decided to adopt this title, I’d be happy to provide access to any available instructor resources. If you’ve chosen a different title or are still considering your options, I’d appreciate a quick update so I can ensure our records are accurate.

Also, please don’t forget to check with your bookstore about any Equitable or Inclusive Access programs that may be available. These programs can provide your students with more affordable digital access to their course materials.

I am sincerely grateful for your attention to this matter; I look forward to the opportunity to hear from you at your earliest convenience.

Cordially,

**
Sales Support Assistant
Higher Education Sales, Americas
**

On one hand, hey, I’m thrilled that they’re promoting our book. On the other hand, it seems like they’re kinda flying blind if they’re sending this promotional message to the author of the book.

“Placebo tests deserve a model, not just a glance.”

Posted on by
1

Miha Gazvoda shares this post with the above title and the subtitle, “Using Bayesian multilevel models to correct bias and calibrate uncertainty.”

He’s using the chickens model from our Slamming the Sham paper in the more general setting of placebo control tests.

In econometrics, a “placebo control test” does not need to literally involve a placebo treatment; it more generally is used to describe a procedure in which the same statistical analysis that was used to estimate a causal effect is applied to a different dataset, or a different part of the existing dataset, in which the treatment did not occur.

For example, if you want to measure the effect of an intervention that occurred in 2021, you could repeat the analysis but using data from a different year. Or if you want to measure the effect on a particular outcome, you could repeat the analysis but looking at a different outcome that should be unaffected by the treatment.

The idea is that, if your estimation method has an artifact or systematic bias, this should show up in the placebo analysis as well, indicating a problem. Conversely, if the placebo analysis does not show an effect, this is taken as evidence that there is no artifact.

In his post, Gazvoda argues that, rather than using the result of the placebo check to make a go/no-go decision, it should be possible to partially adjust the treatment effect to account for the information in that supplementary analysis.

This makes sense to me, and of course I’m happy that he’s using our chickens model.

There’s a tricky thing going on here with the placebo check, which, interestingly, arose in the chicken example too, and that is that we usually don’t have any good theory for where the effect is coming from in the placebo control. After all, if our causal identification is working as designed, we shouldn’t even need the placebo comparison, as we’re already getting an unbiased estimate of the treatment effect. The placebo control is typically there to address unspecified concerns of bias. And, indeed, in practice, researchers don’t always do placebo controls. And when, as hoped, the placebo control shows no statistically significant effect, the inclination is to take that as a reassurance and move on, in the same way that is done with other robustness checks.

The lesson Gasvoda takes from the chickens example is that if replications are available, you can assess the evidence for the placebo adjustments being relevant: you can fit a multilevel model to estimate how much adjustment needs to be done.

From a sociology-of-science point of view, it’s interesting to me that the conventions in biomedical statistics and econometrics go in opposite directions:

– In biomedical statistics, the default recommended behavior is to compute the difference in differences, taking the estimated effect from the main experiment and subtracting the estimate from the placebo experiment. As we explain in the chickens paper, this correction has the disadvantage of doubling the variance of the estimate, a true statistical crime if, as is often the case, the effect of the placebo treatment is indistinguishable from zero.

– In econometrics, the default procedure, if you’re calling it “difference-in-differences estimation,” is the same as above. But if you’re calling it “placebo control,” and the placebo estimate is not statistically significant from zero, the default is to ignore the placebo results entirely, not to adjust for them.

In general we recommend a partial adjustment, with the amount of adjustment depending on the problem at hand. If there is internal replication, as in the chickens example, the appropriate adjustment factor can be estimated from the data. If it’s a one-shot experiment, you’ll need to use prior information. I don’t have any good examples demonstrating how to do that; it’s something we should do.

What gets you is not what you don’t know but what you don’t know you don’t know.

Posted on by
2

I was thinking about the above saying in the context of bad regression discontinuity analyses.

Statistical methods can be characterized in terms of how they can go wrong. Some common modes of “how can things go wrong” include:
– biased measurements,
– selection bias into the observed dataset,
– differences between treatment and control groups,
– nonstationarity or, in general, differences between sample and population of interest,
– biased or noisy statistical estimates,
– missing data,
– problems with functional forms (for example, using a linear model for a nonlinear relation, or not including important interactions),
– problems with error terms (dependence, choice of distribution, etc.),
– unmodeled spillovers, hierarchical structure, or other violations of causal model assumptions,
– researcher degrees of freedom and forking paths,
and lots more!

Regression discontinuity is used for observational studies, and the #1 thing that can go wrong in an observational study is #3 on the above list: differences between treatment and control groups.

The problem is that when researchers perform regression discontinuity analysis, they focus on the problem with the functional form of the expected outcome given the one predictor, often ignoring other potential predictors even if they are screaming to be included (as with age in this example). There’s all this obsessing over the functional form (and I guess Guido and I contributed to this) that’s kind of missing the point (as we discuss briefly here).

So, the problem is not that these users of regression discontinuity don’t know what to do with other predictors; it’s that they don’t know that they don’t know this.

As Tolstoy might have said had he been a data analyst, Models that fit are all alike; every poorly-fitting model is poorly fitting in its own way.

P.S. Here’s some background from the Quote Investigator on various version of the saying that I used as the title of this post.

Why quantitative understanding of effect sizes matters, even if all you care about is the presence of the effect

Posted on by
16

In reaction to my article with Andy King proposing post-publication review, Dan “Fast and Frugal” Goldstein writes:

Your process limits information search, computation, and time so it seems fast and frugal to me. Happy you still associate me with that term. It was something I coined as a grad student.

In your proposal, only hit papers get audited. It reminds me a bit of Mel Brooks’ The Producers in which the protagonists use the logic “who would audit a flop?” and stay under the radar by intentionally producing a bad show. Fraudsters have likely attempted to make their work seem worthy of publication while ensuring it doesn’t attract too much attention. Just like in The Producers, though, this sometimes backfires.

I don’t know about that! My impression with fraudsters is that they think that fraud is normal science, perhaps out of some mixture of bad education in research methods, a view that “everybody does it,” and a general lack of understanding of how non-cheaters (like you and me!) think. Think of people like Wansink who gave general advice to to the world on how to p-hack, or Gino and Ariely, who published papers on dishonesty, or Mary Rosh, who surely believes that whatever shady statistical manipulations she does are nothing compared to the dastardly deeds done by the Democrats.

I’m sure there’s tons of below-the-radar cheating and bad science that we don’t hear about, but a fair number of prominent science fraudsters seem to enjoy the limelight. One reason for this seemingly self-sabotaging behavior, I think, is that cheating enabled these people to attain great professional success for years. They had no reason to think the juice would stop flowing.

To return to my proposal with Andy King: I think it’s ok that only the hit papers get audited. Bad papers that get no intention aren’t doing much damage, right?

Goldstein adds:

By the way, I was just having a conversation about your sensing that something was amiss with the LaCour study. For years now I have used this quote of yours in a talk I give about putting numbers into perspective. I argue that it’s really important that people learn how to put numbers into perspective because if they don’t, they won’t notice that something is unusual and worthy of a deeper audit. You somehow sensed something was up with the Lacour result. You didn’t think it was fraud yet but you knew it was strange because you know how to put such differences into perspective:

A difference of 0.8 on a five-point scale . . . wow! You rarely see this sort of thing. Just do the math. On a 1-5 scale, the maximum theoretically possible change would be 4. But, considering that lots of people are already at “4” or “5” on the scale, it’s hard to imagine an average change of more than 2. And that would be massive. So we’re talking about a causal effect that’s a full 40% of what is pretty much the maximum change imaginable. Wow, indeed. And, judging by the small standard errors (again, see the graphs above), these effects are real, not obtained by capitalizing on chance or the statistical significance filter or anything like that.

My colleagues and I recently wrote a paper on this general topic of average effect sizes. It’s our contention that people generally are way too optimistic about possible effect sizes, in large part because they don’t think about variation. If you ask someone to hypothesize an effect size, you’ll typically get a guess of the largest effect that might occur.

But what if you don’t really care about effect size–you just want to know about the effect?

For example, maybe you don’t believe that women during certain times of the month are three times more likely to wear red or pink shirts, but you are interested in some sort of evolutionary psychology theory of sexual display. In that case, why should the effect size matter? Why care that a study reported an estimate that was ridiculously implausible?

I have two to this questions, and thus two reasons why effect size is important even for problems where you don’t directly care about effect sizes:

1. Effect sizes vary. An treatment that has an effect (that is, a true effect, not just an estimated effect) of 0.1 for one group of people in one setting could have an effect of -0.2 in some other scenario. A treatment effect in an experiment is the sum of all sorts of things, positive and negative, and there’s no logical reason to think the sign of the effect will be preserved. Effect size matters. The issue is not just that a smaller and more realistic effect size is less important; it’s also that smaller effects can be more easily produced by other factors, and this reduces the generality of any claims, even if the experiment at hand was done well.

2. Experiments produce standard errors as well as estimates. If the standard error from a study is large compared to any realistic effect size, then the study contains very little information. Effect size is important in understanding the informativeness of an experiment, and to do this right you need to have some sense of what the true effect size could be. You can’t just use a point estimate from the study itself, as this estimate will inherently be too noisy to use to judge the information in the study. As I wrote in this note for the Annals of Surgery, Post-hoc power using observed estimate of effect size is too noisy to be useful.

Survey Statistics: equivalent models, equivalent weights (locally)

Posted on by
4

Last month we saw that the Times/Siena Poll is now using energy balancing weights (Huling & Mak, 2024). In a toy example, we saw under which outcome models these weighting methods might do well. I was inspired by Little 2004, who saw under which outcome model the inverse-probability-weighted estimator (a.k.a. Horvitz-Thompson) does well. This explains the HT estimator’s poor performance in Basu’s (1971) elephants example (which I’ve used in my post Basu’s Bears).

From Little 2004:

Andrew’s 2007 “Struggles” paper and the 2026 MrPlew paper go the other way: start with an outcome model and back out the weights. These folks all worked pretty hard. I wondered if I could just use Thomas Lumley’s survey package to get the weights. See my posts “struggles with equivalent weights” and my continued struggles. But in the simulation I used a linear outcome model. It would have been more interesting with a logistic outcome model !

The linear case was handled by Andrew’s 2007 paper. The 2026 MrPlew paper extends this by noticing that the linear equivalent weights are derivatives, and using this to define locally equivalent weights, green highlighting by me:

Then the 2026 MrPlew folks use these equivalent weights to compare the target population to the weighted sample, as an MrP diagnostic. (Unlike the Huling & Mak paper on energy balancing weights, the MrPlew folks look at one covariate function at a time, not the entire covariate distribution at once.) One of their examples is from Lax & Phillips 2009 Gay Rights in the States, which I blogged about for pride. In this example, the implied covariate (im)balance for MrP looks not great ! Here’s their Figure 5b:

As Andrew blogged: “It makes sense that implied covariate balance can sometimes be worse for MRP than for raking. MRP is a smoothed version of raking, and unsmoothed raking can overfit.” Yes, but here it’s worse than the raw uncorrected balance ! The MrPlew authors caution here that the locally linear approximation may be extrapolating poorly, and that we shouldn’t over-rely on this model check. They also say that “logistic regression generally balances the variance-weighted covariates, but not the covariates themselves”. I am glad they included this illustrative example and I want to think more about it.

Thoughts ?

Survey Statistics: toy example for energy balancing weights

Posted on by
6

Last week we talked about The Big Changes Coming to the Times/Siena Poll:

  1. New weighting variable: support score = E(2024 vote | other X variables).
  2. New weighting method: energy balancing (Huling & Mak, 2024)

Ben Schneider helpfully blogged about energy balancing as well:

Raking and similar calibration methods are based on balancing means or totals for specific variables…The energy balancing method does something different: it calibrates based on an entire multivariate distribution, as measured by an empirical cumulative distribution function (ECDF).

Jared Huling (of Huling & Mak, 2024) helpfully answered questions in the comments. I’m still puzzling over how energy balancing handles empty cells (unsampled regions of the joint covariate space). I need a toy example.

Consider 2 binary variables, so 4 population cells, with known population shares:

       k=0    k=1    total
j=0    .4     .2     .6
j=1    .2     .2     .4
total  .6     .4

Say the sample is missing folks in cell 11:

       k=0    k=1    total
j=0    .5     .3     .8
j=1    .2     0      .2
total  .7     .3

Consider 4 methods:

1. Classical Poststratification: not defined because of division by 0.

2. Raking: match only the margins. Correct when Y | X1, X2 is additive.

       k=0    k=1    total
j=0    .2     .4     .6
j=1    .4     0      .4
total  .6     .4

3. Energy balancing: minimize the Energy-Distance(F_w, F_pop) between the weighted sample distribution of X1, X2 and the population distribution. Correct when Y | X1, X2 is such that nearby cells have similar means.

Say X1 = young/old, X2 = man/woman, Y = percent Democrats, and no old women are sampled.

Raking is correct when additivity holds: old women = young women + (old men − young men)

Energy balancing is correct approximately when: old women = (old men + young women)/2 ?

library(WeightIt)

pop  <- data.frame(X1 = rep(c(0, 0, 1, 1), c(40, 20, 20, 20)),
                   X2 = rep(c(0, 1, 0, 1), c(40, 20, 20, 20)))

samp <- data.frame(X1 = rep(c(0, 0, 1), c(50, 30, 20)),
                   X2 = rep(c(0, 1, 0), c(50, 30, 20)))

dat <- rbind(cbind(pop,  A = 1),
             cbind(samp, A = 0))

W <- weightit(A ~ X1 + X2, data = dat, method = "energy",
              estimand = "ATT", focal = "1",
              dist.mat = as.matrix(dist(dat[, c("X1", "X2")])))

w <- W$weights[dat$A == 0]
tapply(w, interaction(samp$X1, samp$X2), sum) / sum(w)
       k=0    k=1    total
j=0    .381   .309   .69
j=1    .309   0      .309
total  .69    .309

4. MRP: fit a model for Y | X1, X2. The interaction term’s posterior equals its prior, propagating uncertainty around additivity.

Am I understanding this correctly ?

Survey Statistics: Big Changes in the Times/Siena Poll

Posted on by
13

Yesterday Nate Cohn wrote about The Big Changes Coming to the Times/Siena Poll, with
more details in their poll of Maine.

Say we want to estimate average Platner support in Maine’s likely electorate, E(Y). But we only have survey respondents, R = 1.

The NYT uses survey weights to weight respondents, E(YW | R = 1). In contrast, some pollsters use MRP, fitting a Multilevel Regression model for Platner support, then applying it to the population, E(E_model(Y | X, R = 1)).

Nate discusses 2 Big Changes to how they construct the weights W.

(The polar bear has not yet hiked in ME, but he is training for it. This above is in TN.)

Big Change 1: Support score

A few weeks ago we saw the NYT started weighting on “synthetic 2024 vote”, which is recalled 2024 vote that is validated with the voter file and imputed if needed.

Now they’re also weighting on support score = E(2024 vote | other X variables). Nate explains the motivation:

While a poll can’t weight on dozens of variables, the support score lets us pile a lot of information into a single measure.

This reminded me of the causal inference context, where D’Amour and Franks (2021) “see especially strong performance for propensity weights computed with respect to the prognostic score”, where the prognostic score is E(Y | X, control). In our survey context, this would be a model for Platner support Y. Instead, the NYT use 2024 vote, perhaps for applicability across multiple outcomes Y ?

Big Change 2: Energy balancing

Beyond adding new weighting variables, they’re also changing how they calculate the weights. Nate notes the challenge of weighting on many variables and interactions with typical sample sizes. So they are turning to the R package WeightIt, which implements the energy balancing method from Huling & Mak (2024):

This article introduces a new weighting method, called energy balancing, which instead aims to balance weighted covariate distributions. By directly targeting distributional imbalance, the proposed weighting strategy can be flexibly utilized in a wide variety of causal analyses without the need for careful model or moment specification.

The energy balancing weights do not use outcome Y, but the paper notes that estimates can be improved with a model for Y.

How do energy balancing weights handle the challenge of jointly weighting on many variables with typical sample sizes “without the need for model specification” ?

Structural equation modeling (SEM) and positive definiteness

Posted on by
25

This post is from Bob.

Mitzi and I were swotting up on structural equation models (SEM) for our class this past Monday at the Modern Modeling and Methods (M3) conference at Fordham University. It was a lot of fun and now I think I understand SEM notation. I really like these applied conferences and this was a group of psychometrician, econometricians, and sociometricians. Many if not most of them thought about models in terms of SEM, so we thought we should figure it out. But I was left with a concern you may be able to help me sort out.

The example

The first worked example in Ken Bollen’s seminal 1979 textbook on SEM is a study of how industrialization relates to democracy. It comes from his paper,

  • Bollen, Kenneth A. (1979). “Political Democracy and the Timing of Development.” American Sociological Review, 44(4).

and was reprised in his book

  • Bollen, Kenneth A. (1989). Structural Equations with Latent Variables. Wiley.

I had the pleasure of sitting across from Ken at the invited speakers dinner at the conference, so I’m glad I looked into SEM before that. Good news for the SEM devotees—he released a completely revised guide to SEM a few months ago.

The data and parameters

The data consists of eleven covariates (called “indicators” in SEM) for each of 75 countries. Four of the covariates are related to democracy in 1960 (y1, y2, y3, y4), the same four measurements were taken again again in 1965 (y5, y6, y7, y8) , and there were three measurements of industrialization in 1960 (x1, x2, x3).

The SEM model the original researcher came up with here assumes three latent scalars per country, industrialization in 1960 (IND60), level of democracy in 1960 (DEM60), and level of democracy in 1965 (DEM65). These latent parameters are related in the following way: democracy in 1960 is a regression on industrialization in 1960, and democracy in 1965 is a regression on both democracy in 1960 and industrialization in 1960.

The covariates are then modeled like a seemingly unrelated regression in econometrics. The four democracy 1965 parameters are treated as regressions on the latent level of democracy in 1965, and similarly for the democracy in 1960, and industrialization in 1960.

Rather than independent errors, a SEM model explicitly indicates with arrows which pairs of observations are allowed to have non-zero correlation in the covariance matrix for the observations. The three industrialization observations are assumed to have zero correlation—there are no arrows between any of the three measurements in the SEM diagram. Each of the four measurements in 1960 is assumed to covary with the same measurement taken in 1965. In addition, the second and fourth measurement in each year are assumed to be correlated with each other, which leads to a box-like structure.

The SEM diagram

Here are the arrows in the diagram, where I’m not using their standard LISREL notation, but writing them in R expression syntax to indicate what is regressed on what. In their graphical notation, just replace ~ with <-. All three latent variables and all eleven measurements are indexed by country. 

IND60
DEM60 ~ IND60
DEM65 ~ DEM60, IND60

x1, x2, x3 ~ IND60
y1, y2, y3, y4 ~ DEM60
y5, y6, y7, y8 ~ DEM65

The covariance structure is indicated by stating which pairs of measurements are modeled with non-zero correlation. The first four just pair the measurements of the same thing across 1960 and 1965.

y1 <-> y5
y2 <-> y6
y3 <-> y7
y4 <-> y8

The last pair of correlations are within 1960 and within 1965.

y2 <-> y4
y6 <-> y8

Together, these induce an odd box structure, where y2 is correlated with y6 and y4, both of which are correlated with y8, but y2 and y8 are assumed to have zero correlation.

y2 <-> y6
^      ^
|      |
v      v
y4 <-> y8

Stan implementation

We didn’t get this far in my half of the class, so I will share here the Stan Playground example where I fit Bollen’s example (you can get the data and the Stan model through the Playground link:

It gets the right answer compared to lavaan/blavaan, which is nice. In the Stan code, xi is IND60 and eta1, eta2 are DEM60, DEM65. The relation among the latent parameters are modeled directly as regressions. The correlations among the observations are modeled using soft zeroing, where I just put a tight prior around zero on the structural zero elements, because Stan doesn’t give you a good way of setting up structural zeroes in a covariance matrix (Sean Pinkney or Ben Goodrich might know how to do this?).

This makes me curious how the lavaan package in R manages this. There’s a Bayesian version of lavaan built on top of Stan, blavaan. The first example right at the top of the home pages for both the lavaan and blavaan is Bollen’s democracy model. I guess it’s like the Scottish lip cancer data set for spatial modeling or Fisher’s iris data for regressions.

My questions

Consider a simple diagram among measurements like the following.

x <-> y
y <-> z

This says there can be non-zero correlation between A/B and also between B/C, but the correlation between A/C is zero. It’s a simplified case of the box we saw in the actual example. These arrows implies the correlation matrix looks as follows. 

|        1  rho[x,y]         0 |
| rho[x,y]         1  rho[y,z] | = Omega
|        0  rho[y,z]         1 |

Given that the correlation matrix Omega must be positive definite, this limits the range of rho[x,y] and rho[y,z]. For example, we can’t have rho[x,y] = rho[y,z] = 0.9, or rho[x,z] would have to be greater than zero to maintain positive definiteness.

Q1: Why doesn’t SEM instead say that the correlation rho[x,z] is just the minimum value it can be given rho[x,y] and rho[y,z]? I’m suggesting that we instead treat the above diagram as implying no additional correlation between x and z other than that implied by the correlation between x and y and the correlation between y and z? That is, why try to shrink rho[x,z] all the way to zero? From the text, it feels like the motivation is to enforce zero correlation in the model. But all this is doing is simplifying regressions—it won’t actually enforce zero correlation among the measurements that are modeled with zero correlation. I wished I’d asked Ken this question at dinner, but I’ll ping him about this blog post and hopefully get a response.

Of course, in the pragmatic Bayesian workflow, we’d use posterior predictive checks to evaluate whether there’s unmodeled correlation between x and z.

Q2: I’m also curious what Andrew and others think about enforcing structural zeroes in correlation between measurements as opposed to just estimating a dense covariance matrix and inspecting where the correlations fall.

Ph.D. student opening in Sweden on Earth Observation, Data Science, and AI for poverty estimation

Posted on by
1

Adel Daoud writes:

I’m writing to ask for your help circulating a PhD opening in my group at Chalmers, the AI and Global Development Lab (www.aidevlab.org). The position is in Earth Observation, Data Science, and AI for poverty estimation, the Data Science and AI division (Department of Computer Science and Engineering). We are looking for candidates with a strong grounding in data science, computer science, deep learning, statistics, or similar— remote sensing experience and causal inference are welcome bonus.

Ad and application portal: https://www.chalmers.se/en/about-chalmers/work-with-us/vacancies/?rmpage=job&rmjob=14818&rmlang=UK
Deadline: 20 June 2026.

Here’s the description of their center:

The AI & Global Development Lab fuses AI with Earth Observation to illuminate the causes and consequences of human development across time and space.

Our interdisciplinary team, comprising data scientists, computer scientists, and social scientists, develops methods to better understand the multi-scale dynamics of pressing global issues, including poverty, conflict, sustainability, and the effectiveness of policy interventions.

By analyzing satellite imagery from 1984 to the present, AI search agent swarms for large-scale knowledge discovery, and other planetary-scale sources, we are reconstructing historical and geographical development trajectories at a level of detail never before possible, working to offer new insights into the changing face of development worldwide.

We also invite you to visit PlanetaryCausalInference.org for more information about the causal arm of our project.

They call it “Planetary causal inference,” which seems to fit the themes of this blog.

To what extent is it true that “All intelligence, human or artificial, must extract structure from correlational data”?

Posted on by
30

Someone pointed me to this article, “Does AI already have human-level intelligence?” You can click through to read the whole thing; spoiler alert: their answer is Yes.

I don’t have much to say about the main argument of the article–it’s a topic we’ve gone over all too much in past comment threads–also, as non-user of chatbots, I’m really the worst person to ask for an opinion on the topic. Indeed, the other day I was contacted by a reporter for a story about “vibe analytics” where people use chatbots to write code to perform data analysis. I shared my thoughts for a few minutes but then referred the reporter to Bob and Jessica, as they both have thought a lot more about this than I have. I continue to (a) think that it can make sense to consider chatbots and ping-pong playing robots as having human-level intelligence, and (b) agree with Gary Smith that it remains a big problem when people think chatbots have a level of understanding that they don’t actually have. But, again, my thoughts on this shouldn’t count for much.

But there is one thing in this new article that I did want to comment on. It was just an aside, not the main point by any means, but interesting:

“All intelligence, human or artificial, must extract structure from correlational data.”

Is this true? I don’t know about that, for two reasons. First, I can’t think of many cases where I (that is, my human intelligence) have extracted structure from correlational data. Setting aside my professional life as a statistician and social scientist, when have I done this? I’m not sure. Yes, I’ve estimated parameters from correlational data–for example, if I’m playing sports I make inferences about the abilities of other players based on what they’ve done on the field in the past. But that’s not structure, exactly. There is structure in the world, like the difference between cats and dogs. You can dress a dog up like a cat but it’s still a dog. Essentialism and natural kinds and all that. But that’s not anything I extracted from correlational data: I know it because people told me.

One way that I’ve extracted structure from correlational structure is that as a kid I heard lots of talking and read lots of books and I extracted lots of structure of the language from that. But that’s just one example–an important example, sure, but I don’t know that it’s a characteristic of “all intelligence.”

Another way to look at this is that, as a community, we’ve extracted a lot of structure in the world–it’s called doing science–and some of this is from correlational data (Kepler figuring out planetary orbits, Galton and his table of heights, etc.) but lots of the structure we’ve extracted comes either from logical reasoning (Newtonian mechanics, relativity theory) or from experimentation–they say Galileo did a bit of that.

This doesn’t invalidate the argument made in the linked article–after all, there’s no reason a computer program can’t do pure theory or conduct experiments–; I just thought it was interesting. Speaking in some fundamental sense, it seems to me that experimentation, not just observation, is a crucial part of how we often extract structure. We experiment a lot when speaking. On the other hand, sometimes, as with Kepler or with someone learning a language from reading books, the information is all, or almost, correlational.

It’s an interesting thing to think about. We could throw this at a chatbot and see what it would say–or, more precisely, we could see what it could extract from what humans have said about related topics. But humans have said a lot; it’s a mark of intelligence to be able to read a million books and then extract their key points.

P.S. After reading a bunch of comments, I realize that I kind of missed the point of the passage I was quoting.

My argument above is that intelligence doesn’t learn about structure only by extracting structure from correlational data. Intelligence also learns about structure from logical reasoning and experiment.

But my argument doesn’t refute the quoted line, “All intelligence, human or artificial, must extract structure from correlational data.” That quote doesn’t posit that intelligence only learns from correlations. It just says that learning from correlation is part of the mix, and I agree with that.

So, as long as that passage is interpreted as saying that “extracts structure from correlational data” is necessary for “intelligence,” I’m ok with it. My problem was my interpretation (or misreading) that correlational analysis was sufficient.

“The Data Analyst’s Guide to Cause and Effect”

Posted on by
17

Theiss Bendixen and Benjamin Grant Purzycki wrote this book. He writes:

The website holds:

– All data and code used in the book
– Free sample chapters
– Bonus material

These aren’t quite the same methods for causal inference that I’m inclined to use (for my own approach, see chapters 18-21 of Regression and Other Stories), but their presentation is clear and has code, and it’s always good to see another perspective.