This series has used a few survey statistics textbooks, see my list below. Which of your favorites am I missing ?
Särndal, Swensson, Wretman 1992:
Survey Statistics: favorite textbooks ?
11
Category Archives: Teaching
Columbia University says, “We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA.”
Claire Shipman, the president of Columbia University, writes:
We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA. I intend to uphold that tradition for several reasons.
First, as a nation, we should be able to tolerate ideas, facts, and opinions that we don’t like. Tolerance is essential to building resilient individuals and a resilient democracy.
Second, it’s our mission. We have a responsibility to guide students toward intellectual rigor, to enable them to grapple with, and examine carefully, ideas different from their own. More broadly, a willingness to test assumptions is essential to our academic excellence in every field.
Third, and perhaps most important, universities—Columbia in particular—are uniquely positioned to help change the national tone. We can guide and model the art of examining complex, potent issues in a deliberate, respectful fashion. . . .
Our instinct might be, in such an era, to shy away from challenging events and speakers. The better course is to model how to do it well . . . Our statutes lay out quite clearly and sensibly that expression, on our campuses, does have limits, in particular as regards harassment, discrimination, or the threat of violence. Peaceful protest, civil discourse, a respectful exchange of ideas—this is our inheritance at Columbia, and I want to ensure that it is also our legacy. . . .
So, as this week begins, and as we continue to debate the affairs of the world and this nation on our campuses, consider this: at times there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it. Maybe even seek it out. Such a practice makes us stronger.
I agree. And, although our president did not say so in this message, this policy of protecting and welcoming dissent, even when it is controversial or disagreeable, runs counter to some actions of our university in recent years:
- The delayed reaction to the sexually-assaulting doctor in the medical school, without any seeming consequences to the people at the university who enabled this behavior.
- The fake U.S. News numbers, where, again, there seems to have been very little interest by the university administration in finding out who did it and why. As with the medical school scandal, the avoidance of any serious investigation does not seem consistent with the goal of promoting free speech.
- Barnard College, a division of Columbia University, attempting to cancel the showing of a controversial film on campus. At that time, it seemed that everything was about avoiding offending anybody and about avoiding any controversy—really the opposite of the view that “we should be able to tolerate ideas, facts, and opinions that we don’t like.”
- Barnard College “mandating that students remove any items affixed to room or suite doors” under the rationale that they “do what we can to support and foster the respect, empathy and kindness that must guide all of our behavior on campus.” Empathy and kindness are fine, but, again, it contradicts the idea that “there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it.”
As I noted the other day, Barnard College has also announced a new policy promoting free expression, and I think it’s a good thing.
I completely agree with the president of Columbia that “we should be able to tolerate ideas, facts, and opinions that we don’t like,” and I completely agree with the president of Barnard that “Colleges and universities need to reconfirm our commitment to nonviolent forms of disagreement — even when we are confronted with voices that disparage or dismiss identities and worldviews. This is also a time to foster more disagreement, not less.” I was getting very frustrated with all the happy talk and the official attitude that we shouldn’t be disagreeing. The university—and, of course, the country—has people who disagree with me on important issues, and people I don’t respect at all, for moral, intellectual, or political reasons. And I guess lots of those people don’t respect me either! Which is too bad, but we still have to exist, and we should have the right to express our views, both publicly and privately. We shouldn’t be fired for our nonviolent acts (except to the extent that they interfere with our ability to do our jobs), nor should be discouraged by the university from expressing them.
And, just to be clear, no one at Columbia has ever suggested or even requested that I keep my mouth shut in class or outside of class, including when I’ve criticized the government or the Columbia administration or anything else. I have nothing to complain about on my end, but I do think it was a mistake for them to crack down on dorm room doors, to try to stop the showing of controversial movies, and to keep quiet the bad behavior regarding the fake statistics and the terrible actions at the medical school.
I’m happy about this new policy and I hope the administration follows up by changing the dorm room door policy, apologizing for trying to suppress the movie, and starting and publicizing independent investigations into the U.S. News numbers and the enabling of the sexual assault at the hospital.
Also let me emphasize that by bringing up these earlier cases, I’m not criticizing the Columbia president Shipman, as these things were done before she took office. I also understand the desire to move forward rather than looking back; still, I think that a great way to reinforce the policy of enabling difficult conversations is to revisit some recent examples where the university had unfortunately avoided doing so.
“It’s in our DNA.”
One thing that jumped out at me when reading the president’s email was the statement that freedom of expression, especially in difficult conversations, is “in our DNA.”
My first reaction was the annoyance of the persnickety social scientist: calling it “DNA” is just wrong, and it indeed muddies the waters, as we’re talking about something social, not genetic.
But that reaction of mine was silly. Obviously no one when using “DNA” in that sense is taking it literally; it was a metaphor. And maybe we can gain some insight by taking this metaphor seriously.
For something to be in an organization’s “DNA,” that would imply that it’s not just a transient trait; rather, it has to be in some equivalent of self-replicating code. The idea is that freedom of expression and willingness to confront uncomfortable ideas are not just attitudes that sometimes has existed at Columbia, but rather that they are embedded deeply enough that they will persist. Indeed even if free inquiry and expression at Columbia are suppressed (as with the sexual assault, the fake statistics, the dorm-room censorship, and the attempt to suppress the film screening), the self-replicating aspects of our university’s rules and culture will cause them to return.
And, from that perspective, I do see something in our president’s analogy to DNA. Columbia has for many years celebrated the legacy of the 1968 protests, while recognizing that the other side at the time had legitimate concerns too. That is, Columbia has recognized the value of protest without implying that any particular protest action has to be recognized as being a good idea. We have the University Senate, which dates from that period, as well as generations of new students and staff who have exercised the freedom of dissent—even dissent that some can find disagreeable—and recognize this as a Columbia tradition.
Indeed, this recent statement by the president can be viewed as a consequence of that DNA, an expectation that our university is a place with diverse enough attitudes that there will be people who disagree with and even despise each other but still have the right and, at times, the obligation, to air these uncomfortable disagreements publicly.
Because the alternative, of simulating a nonexistent harmony, would be intellectually dishonest and politically dangerous.
Survey Statistics: 2 different uses of simulated data
Last month Andrew wrote:
I use simulated data to understand how a model works; it can be viewed as a form of mathematical analysis, a way of working out the implications of a set of assumptions.
I don’t use simulated data to learn about the external world. For that, I need to take measurements of reality.
Let’s start with the latter, which concerns using chatbot-simulated survey responses.
Simulated data to learn about the external world:
Andrew wrote: “the chatbot is trained on the past, but the usual reason for surveys is to learn about changes”. Pew Research agrees: “Survey-taking is best left to humans”. And a year ago we saw that Thomas Lumley agreed: “It [Claude or ChatGPT] will probably do this better than I could, but it’s not magic.”
That said, Andrew suggested using the simulated response as an additional adjustment variable. Jessica Hullman wrote last year about a suggestion from Broska, Howes, and van Loon (2025) that uses LLM predictions as auxiliary data in something like a survey regression estimator. Andrew of course suggests Multilevel Regression and Poststratification (MRP) to incorporate auxiliary data like LLM predictions.
Now back to the first use of simulated data.
Simulated data to understand how a model works:
Andrew has written lots about this. I teach it in my course with Dr Arjun Potter at the Nelson Mandela African Institution of Science and Technology (NM-AIST). Our example is Potter et al. (2026), who study the effects of drought, fire, and herbivory on growth of various acacia tree species. In our course, we simulated data ahead of real data collection to understand the model, e.g. how block effects change the precision of the fire effect. Students understood that this exercise does not replace real data collection. We are hoping to teach these ideas again in the new year (funding dependent). For the past three iterations, see “Hand-drawn Statistical Workflow at Nelson Mandela” and “Survey Statistics: connections to experimental design”.
“Phonics is a fad, but it’s also good, so we should do it anyway, even though it will probably disappoint people”
Tyler Watts and Drew Bailey write:
Educational policy is often viewed as a field of fads, where one follows another for seemingly arbitrary reasons. We might like to believe that fad-based policy could be superseded by evidence-based policy, with rational actors choosing policies because of scientific evidence. But that’s been hard to achieve.
In the cynical version of the story, hucksters overpromise on their pet projects, we make the changes they advertise, and they get rich while the public is left disappointed. This probably happens more often than it should, but this narrative ignores the conditions that make such disappointment ubiquitous in educational policy.
Why? They explain:
Even if the hucksters all disappeared tomorrow, we’d still have this hype cycle, because overpromising seems inevitable when effects are small, and political disappointment will always follow. . . .
The educational policy research ecosystem usually selects for safe, often evidence-based, politically feasible, but ultimately oversold ideas, and we think that hurts our ability to make progress. In education, real effects are really small. . . .
Why are the effects so small? The outcomes we like to measure, such as reading comprehension, depend on many personal and environmental factors, making it hard for a discrete change in schooling to make much of a difference. . . .
This is a well-known problem in education research. For example, it came up in my 2018 talk at the Society for Research on Educational Effectiveness, and the point wasn’t new then either!
But, as Watts and Bailey explain,
The extreme education-nihilist position—that educational improvements can never affect students’ skills—is actually wrong. Endorsing it is likely to lead to poor education policy. The edu-nihilist position overlooks that small effects are not always zero effects. Small effects can actually be meaningful and, in some cases, worth it from a cost-benefit perspective for society.
One way to think about this is to consider classroom teaching.
I work my butt off every semester teaching my classes. Every class takes a lot of effort, both by me and the students. But any given class can’t have much of an effect on total learning. Indeed, the entire semester doesn’t have such a large average effect, when you consider that some students already could do most of it before the semester began, some still don’t get it when the semester is over, and others learn some things which they later forget. (On the plus side, students can learn all sorts of useful things that don’t happen to be on the exam.) But the effect of these classes is not zero. Students take them and they learn some things. So of course there’s no simple switch that will improve performance by 0.2 standard deviations. We’re already doing what we can. And the flip side of this is that a small improvement is not nothing. We teach each class knowing that at best it is giving a small average improvement, but it’s still worth doing.
Watts and Bailey continue:
When education policy researchers say something “works,” most people assume that effects are much larger than they are. . . . In most cases, researchers and policymakers fall prey to effect size magnification because they develop real sympathy for a pet program. When this happens, they put forward the most promising evidence, which is likely biased. The field then develops unrealistic expectations, and researchers, policymakers, and advocates downplay or withhold information about their uncertainty.
Hey, here’s an example from celebrated economist James Heckman, who doesn’t have much of an understanding of selection bias and goes around promoting wildly exaggerated estimates of early childhood programs.
Here are Watts and Bailey on what happens next:
Educational policies are uniquely poised to fall apart when overhyped, because public scrutiny of educational outcomes is so intense. Because most of the variance in student achievement comes from factors schools can’t control, overselling sets up an inevitable letdown. When disappointing results emerge and large gaps in achievement remain, we overreact to the apparent failure. Feeling the heat, the researcher, policymaker, or advocate might retreat and say that whatever was promised was never supposed to be a “panacea.” This always feels cheap because the language rarely shows up in the press releases, grant proposals, white papers, or speeches preceding the policy. Ultimately, we are left abandoning ship, even if the policy in question was actually helping.
The clearest example of this is probably the accountability reforms pushed by No Child Left Behind. Everybody hates No Child Left Behind because it was terrible and didn’t work! Right?
Actually, our best policy evaluations suggest it did work; it just had small effects, which is exactly what we should have expected. Instead of taking the small effects as a sign that something is working and figuring out ways to improve it, politicians and advocates decided it was a failure and moved on to other things.
And then they turn to the latest trend:
In a country that can’t agree on much, states from Alabama to California have adopted policies designed to promote the science of reading. . . . Count us as phonics believers. Kids need to know how to sound out words when they learn to read. We taught our own kids how to read this way, and we ourselves even learned to read this way in the pre-science-of-reading dark ages of the 1990s. But the unfortunate reality is that we have no reason to believe a sudden shift to a phonics-based curriculum will significantly affect reading outcomes in this country.
There are several reasons why.
First, meta-analyses of phonics interventions show fadeout. This means that kids who get phonics instruction for some period of time look a lot like kids who didn’t get phonics instruction in terms of reading just a year or two after the intervention ends.
Second, teaching phonics seems to help with decoding and word recognition. This is an important building block of literacy, but there isn’t much evidence that it helps with eventual reading comprehension—the ultimate goal. . . .
Finally, we are fooling ourselves if we actually believe that phonics hasn’t been used in classrooms up to this point. Do we really think that it never occurred to teachers in the “woke” states to help kids break words into syllables and sound them out? The counterfactual condition here has become a strawman. . . .
The best evidence on what happens when states adopt early literacy policies is probably this working paper, which finds effects of up to about .10 standard deviations . . . on high-stakes tests for 3rd graders (that’s good). They didn’t find much on low-stakes tests administered to 4th- or 8th-graders.
So, if we enter a halcyon period for phonics instruction, we might help a lot of kids learn basic decoding skills when they are first learning to read. But . . . effects will be smaller than we were led to believe. Importantly, they probably won’t move the needle much on outcomes like NAEP trends, which compare U.S. cohorts of 4th and 8th graders to each other. . . . The question is: how will we interpret this when the sobering news starts to come in?
Just in case we weren’t clear: doing more phonics in early-grade classrooms is, for the last time, probably a good thing.
They conclude:
Education policy is not the only field that faces conflicts between public and expert opinion. But the combination of big expectations, small effects, and constant attention to children’s educational outcomes means education policy is likely to keep experiencing these conflicts.
What can we do to mitigate this educational political economy problem? . . .
We should ask interested parties to make predictions on the record about how these children will differ (and by how much). And our society should reward researchers and policymakers whose mental models are closer to reality, rather than those who are best at selling potential solutions in the first place.
And my own thoughts
I agree with everything that Watts and Bailey wrote above, and also have a few things to add:
First, there’s my recent paper with Amy Krefman, Lauren Kennedy, and Jessica Hullman, Hypothesizing an effect size by considering individual variation, where we try to think systematically about how to get more realistic expectations of average treatment effects.
Second, a point from my above-linked SREE talk, that research should be less isolated from practice. From one direction, research studies should be more realistic, more connected to the real world of education—this is a point that Watts and Bailey make in their post. From the other direction, general practice should follow principles of research: students, parents, teachers, and administrators should be empowered to make choices and experiment, and, along with this, choices and outcomes should be measured so that effectiveness can be studied. We should all be trying to do better anyway, and we should design our data collection so we can all best learn from these experiences. As we discussed in a different context, local data collection, collaborative analysis, and local recommendations.
Third, ummm, this is a longer one so I’ll just quote myself:
You can conceptualize an education intervention as a vector, where the direction of the vector is the material being learned and the length of the vector is the amount that students are motivated to work. You want the material learned to be useful—you’d like the vector to have a positive “dot product” with the vector of skills, knowledge, and understanding that will be useful going forward—but, conditional on those two vectors being roughly aligned, the real gain is in the magnitude. And this magnitude will be an interaction between the teacher and student: there’s no button to push to create motivation, and if there were a button it would already have been pushed.
What I’m saying is that, when thinking about acupuncture, or physical therapy, or coaching, or teaching, we have to go beyond what I’ve called the penicillin model of science, the idea that innovations come from nowhere and that the job of statistics is to design and analyze experiments to reject the null hypothesis of no effect, and in which the treatment in such experiments is considered as a black box, with the goal being to estimate an average treatment effect.
I don’t think the penicillin model usually applies. Most of the time in health, education, and just about any field, improvements are incremental, and the goal is to improve the process while gaining understanding. Clinical trials and offline experiment both play a role, and you’re not going to learn much by studying a treatment as if it’s a black box.
This is not to say that there cannot be new developments in any of these fields, nor is it to deny that such developments can sometimes arise serendipitously. I just think that, in any case, you have to go beyond the average treatment effect and think about the mechanism of action.
“Why did ANOVA fall out of fashion?”
A student asks the above question.
My response: Anova is still important; it’s just been subsumed by hierarchical models.
The link is to my 2005 paper, Analysis of variance: Why it is more important than ever, which begins:
Analysis of variance (ANOVA) is an extremely important method in exploratory and confirmatory data analysis. Unfortunately, in complex problems (e.g., split-plot designs), it is not always easy to set up an appropriate ANOVA. We propose a hierarchical analysis that automatically gives the correct ANOVA comparisons even in complex scenarios. The inferences for all means and variances are performed under a model with a separate batch of effects for each row of the ANOVA table.
We connect to classical ANOVA by working with finite-sample variance components: fixed and random effects models are characterized by inferences about existing levels of a factor and new levels, respectively. We also introduce a new graphical display showing inferences about the standard deviations of each batch of effects.
We illustrate with two examples from our applied data analysis, first illustrating the usefulness of our hierarchical computations and displays, and second showing how the ideas of ANOVA are helpful in understanding a previously fit hierarchical model.
This is the paper that discusses the five different definitions of fixed and random effects from the literature; scroll down to page 20.
I guess the problem is that Anova got associated with clever-but-ultimately-bad ideas of null hypotheses and F tests. Hierarchical modeling remains very important; I think it’s an underrated topic in statistics.
Bayesian Workflow free pdf!
Our wonderful new Bayesian Workflow book is now available as a free pdf! Just go the link—it’s right there!
I recommend getting the hard copy too because you’ll want to be able to read it while working on the computer, and the cost of the book is trivial compared to the benefit from faster learning that you will get by being able look at the book without taking up valuable screen real estate; also you can see connections when flipping through the pages that might not be apparent by viewing one page at a time on a screen.
Conversely, if you have the hard copy, you should still download the pdf because it fixes a bunch of minor errors that we caught after the book went to press. Also in the printed version we accidentally repeated some of the exercises in chapters 2 and 3. For the pdf we fixed this.
Regarding the content, as I wrote last month, with Bayesian Data Analysis, the big steps forward were:
- Going beyond Bayesian inference to also consider Bayesian model building (as a researcher, you construct the model, it isn’t just given to you as in a textbook), model checking (breaking through the absolutely horrible attitude, common to Bayesians in the early 1990s, that the model was “subjective” and thus should not be checked), and model improvement (continuous model expansion, not the misguided idea of assigning posterior probabilities).
- Going beyond simple conjugate models. BDA had lots of hierarchical models, also lots of computational tools so that you could fit the models you want by putting them together from understandable components. And I like how we had a clear separation between modeling and computing. The model comes first, then you figure out how to compute it. Or you set up a model that works within your computational constraints.
- A Bayesian approach to sampling and causal inference. This was Rubin’s framework in which unobserved units in the population and unobserved causal outcomes are treated as missing data and are part of a joint probability model. We worked this out in chapter 7 of BDA (which became chapter 8 in the third edition of the book).
- Lots of live examples. Not just “real-data examples,” but problems we’d directly worked on. This motivated us and I think it gave our readers a sense of how Bayesian methods worked not just in theory but in applied problems.
- A pragmatic view of probability as a measurable quantity. That’s right there in chapter 1. Bayesian methods are not the product of a philosophical stance; they’re a way to connect models and data using probability.
I could go on and on, but for that I can refer you to the Bayesian Data Analysis book.
And these are the key innovations of Bayesian Workflow:
- Going beyond Bayesian data analysis (model building, inference, model checking, and model expansion) to consider the larger process of statistical modeling, including comparisons of multiple models fit to a single dataset.
- A fuller use of informative priors. This is a big deal. In BDA we still had a bit of the Bayesian cringe going on. One reason we’ve moved toward stronger priors is that the replication crisis has taught us that the amount of prior information available in any given problem is often approximately the same as the information coming from an experiment (see here, for example). Informative priors also fit our increased focus on generative modeling, and we’re doing a lot more prior predictive checking to understand the implications of our models.
- More integration between modeling, data analysis, and computing. One way to see this is that the Bayesian Workflow webpage has the code to run all our examples. We also have lots of code snippets in the text as a way of demonstrating the way in which coding is central to our statistical workflow.
- Lots more live examples. It’s been 30 years since BDA first came out. One reason that Bayesian Workflow has 11 authors is that different collaborators worked on different examples (but the three principal authors read through the entire book, so the general approach should remain coherent).
- Simulation-based experimentation. This is something my colleagues have been doing more and more over the years. At its most basic, simulation-based experimentation provides a best-case baseline for statistical methods: if you can’t recover your quantities of interest with sufficient accuracy under ideal conditions (when your data are simulated from the model you’re fitting), then you know you’re in trouble. And often this is the case! Beyond that, we can simulate from one model and fit another, and see what happens. Simulation experiments aren’t always so easy to construct, as they involve specifying the entire data-generation process. But we think this is effort worth expending, as it involves thinking about the problem you’re working on.
I could go on and on, but for that I can refer you to the Bayesian Workflow book.
What contributions can academic statisticians make to sports analytics? (a discussion related to the launch of the new open-access Journal of Statistics and Data Science in Sports)
Related to our recent post on the absurdity of open access fees, somebody recently informed me that a group of statisticians that work in sports decided they were sick and tired of this with the Journal of Quantitative Analysis in Sports and launched a new open access journal, the Journal of Statistics and Data Science in Sports:
JSDSS was founded on three core principles.
First, our commitment to open access is more than just lip service. At JSDSS, open access means free to read and free to publish—no exceptions. JSDSS is a Diamond Open Access journal. Research supported by academic institutions, public funding, or personal effort should not require a payment to reach the audience it deserves.
Second, reproducibility is essential and not an afterthought. The credibility of sports analytics research depends on our ability to verify, replicate, and build upon earlier work. JSDSS will actively incentivize transparency in data, code, and methodology, and work that meets our reproducibility standards will receive a special designation recognizing this commitment.
Third, we believe sport is a rich and underutilized laboratory for statistical and data science innovation. From player evaluation and in-game strategy to league design and fan engagement, sports data present compelling, real-world problems that demand rigorous and creative analytical thinking. We intend for JSDSS to be the definitive venue for this work.
Cool! I should send them something. We think about statistics and data science in sports a lot around here.
My only concern is that I get the impression that the cutting-edge work on sports analytics is happening outside of academia. So I hope that this new journal can get useful contributions from people who are working in sports analytics who have material they can share without compromising their competitive advantage.
What contributions can academic statisticians make to sports analytics?
Or we could flip it around and ask, What contributions can academic statisticians (like me!) make to sports analytics? Here are a few things:
– Developing general methods that can then be used in sports analytics (as here);
– Writing textbooks explaining general methods that can then be used in sports analytics (as here);
– Teaching students who can then work in sports analytics, or consulting on sports analytics projects, which can be thought of as a form of intense teaching;
– Doing work in sports analytics which, although it is not cutting edge, can still give valuable insights (as here);
– Evaluation and criticisms of published work in sports-related topics (as here);
– Contributing to the sports analytics community (as here, and indeed as in the present post);
– Collaboration on sports strategy, or sports medicine, or the sociology of sports, or various other places where sports links up with academic research;
– Clearing up confusion on topics related to statistics and sports (as here), along with social-sciency thinking about how these misconceptions persist;
– Sports-related research where it can be helpful to have an outside perspective, something available to academics who aren’t on a deadline (as here).
There are probably some more things I didn’t think to include on this list.
Frequentism for Bayesians: He wants to teach frequentist methods to engineering students with a strong Bayesian background
Beyond the teaching question, this is an interesting topic on its own: thinking about classical statistical ideas of point estimation, hypothesis testing, and uncertainty quantification, but taking Bayesian methods as a starting point.
The idea is that, instead of structuring this based on the methods of maximum likelihood estimation, null hypothesis significance testing, and confidence intervals, you start with the goals of estimating parameters, making predictions, comparing and evaluating models (i.e., alternative explanations of the world), and summarizing and working with uncertainty.
If you’re interested in using Bayesian methods, we have two books (Bayesian Data Analysis and Bayesian Workflow) on the topic. But with these methods under our belt, we can now go back to the motivating questions–the more general statistical goals, which exist without reference to any particular models or any particular Bayesian or frequentist methods–and consider them from scratch.
This seems important.
My discussion here is motivated by this question sent in by Opher Donchin:
Here’s one for your blog (although I’m happy to have your take as well).
I’m transitioning our undergrad biomedical engineering course this year from a standard frequentist syllabus to a Bayesian approach. We are mostly following the first 6 chapter of Bayesian Analysis with Python by Osvaldo Martin. In order to get the department to agree to this, I had to promise to teach basic frequentist methods.
Thus, I need to teach frequentist methods to students with a strong Bayesian background. There doesn’t seem to be much available on this. There is of course material comparing the two approaches, but I mean specific material designed to explain the frequentist approach to someone who knows the Bayesian approach.
I’d love to know if anyone knows of resources or has experience of insight or advice.
I replied that I’ll see what the blog commenters suggest, but in the meantime, I recommend chapter 4 of Bayesian Data Analysis as a start.
Donchin responded:
Yes. Chapter 4 is very good on the principles involved. Much of it addresses the Bayesian alternatives to frequentist procedures or the Bayesian perspective on them.
I’m wondering about something more concrete, aimed at a less sophisticated audience. That is, my students will know how to build models, how to interpret the posterior samples, the basics of Bayesian workflow and also model comparison. On the other hand, they will have no knowledge of confidence intervals, maximum likelihood estimates, hypothesis testing, or multiple comparison procedures. Reasonably, my department demands that they be able to read the biomedical literature where such terms are widespread.
I want to give them an understanding of frequentist procedures without getting bogged down in frequentist justifications.
To take an example, I want to explain what an F test calculates when understood within a Bayesian framework. To that end, I can show students a Bayesian model of a normal distribution of group means and a normal likelihood within each group. Then, I can work through what a Bayesian would need to calculate on that model to produce an F statistic and an F test. It has something to do with summaries of posterior distributions of ratios of variances.
What I’m hoping for is somewhere where such questions are worked through in detail so that it could be used for developing lectures.
Of course, chapter 4 of BDA is 10 years old at this point. I imagine some of the ideas may have developed since then.
Ahhh, good point! Chapter 4 of BDA is for statisticians who already know the classical methods and want to understand how these can be understood in light of Bayesian principles and adapted within a Bayesian workflow. But it’s really a completely different task to explain classical methods to students who haven’t already learned them. The idea would be to retcon ideas of classical statistics from a Bayesian angle.
This would be worth doing.
In the meantime, I recommend . . . chapter 4 of Regression and Other Stories, where we go through basic principles of point estimation, hypothesis testing, and uncertainty quantification from an applied perspectives. Also, if you flip through that book, you’ll see other places where we discuss classical procedures from first principles. We don’t have any F tests or multiple comparisons adjustments because I can’t bring myself to care about those things, but a lot else is there, so you might be able to put together much of what you need from that book.
In response to that recommendation, Donchin wrote:
I agree that Chapter 4 of Regression and Other Stories touches on many of the important points, but it is not sufficient for my needs.I am trying to teach a Bayesian-first undergraduate statistics course to biomedical engineers that also gives a background in frequentist approaches allowing them to function effectively in environments that require them to use or understand frequentist stats.This means that the frequentist-realted topics we cover include:
- Estimation: MLE, standard errors, confidence intervals
- Hypothesis testing: p-values, Type I/II errors, power
- Proportion tests and t-tests (independent and paired)
- Effect sizes; multiple comparisons (briefly)
- Linear and multiple regression: least squares, coefficient tests, R2, F-tests
- ANOVA: categorical predictors, interactions, sums of squares, effect sizes
- Pearson correlation and inference
- Repeated-measures and mixed-effects models
I don’t think I could bring myself to teach a lot of the above topics, except in an “inoculation” sort of way, but I recognize that many students will need it, so if anyone has some good suggestions for Donchin, just leave them here in the comments!
Answering some questions about statistics from a high school student
This came in the mail:
My name is ** and I am a junior at ** High School. I am writing to you as a part of a summer assignment for my upcoming research class where I had to choose an expert in the math field to contact. I chose you considering your work and investment in the statistics field and how you have used statistics in the political landscape. I found it very interesting and I was wondering a few things,
• What got you into math and statistics?
• What advice would you give to students interested in statistics?
• What is the biggest challenge when working with statistics?
• What is a common mistake people make when looking at statistics?
• Where would be good resources to learn more?
I appreciate your time and I hope to hear from you soon.
Here were my responses:
• What got you into math and statistics?
See here: https://statmodeling.stat.columbia.edu/2010/11/02/fragment_of_sta/• What advice would you give to students interested in statistics?
In addition to statistics, you should have some applied field you are interested in. This could be biology, economics, sociology, medicine, business, communications, history, . . . just about anything. Then when you learn statistical ideas, consider how they apply to this field, and work on a project applying statistics in that way.• What is the biggest challenge when working with statistics?
Measurement; see here: https://statmodeling.stat.columbia.edu/2015/04/28/whats-important-thing-statistics-thats-not-textbooks/• What is a common mistake people make when looking at statistics?
Looking at the numbers and not checking where they came from. For example, see here: https://statmodeling.stat.columbia.edu/2017/01/02/bogus-north-korea/• Where would be good resources to learn more?
I recommend our blog: https://statmodeling.stat.columbia.edu/ and our book, Active Statistics, which is full of stories: https://sites.stat.columbia.edu/gelman/active-statistics/
Also I like Frederick Mosteller’s classic book from 1965, Fifty Challenging Problems in Probability. I was a teaching assistant for Mosteller for the last course he ever taught!
Since I was answering the questions anyway, I thought I’d post it all here as others might be interested too. I think that all of this is reasonable advice.
I recommend this book on Applied Regression and Causal Inference
This post offers two recommendations and a story.
First, the recommendations.
• Here’s the book referred to in the title of the post. I highly recommend it! My only regret is that we didn’t call it Applied Regression and Causal Inference, cos that’s what it’s about.
• I also recommend our book, Active Statistics, which has literally hundreds of stories, class-participation activities, computer demonstrations, and discussion problems. A great teaching resource and also just a good read. Really!
Now, the story.
This came in the email one day:
Dear Professor Andrew Gelman,
I hope this message finds you well. I’m part of the Cambridge University Press sales support team and I work with your local Cambridge Sales Representative ** **@cambridge.org
We wanted to follow up regarding your course(s) for Fall 2025 and ensure you’ve had a chance to review Regression and Other Stories, 1 ed by Andrew Gelman/Jennifer Hill/Aki Vehtari as a potential option for your course materials.
If you’ve decided to adopt this title, I’d be happy to provide access to any available instructor resources. If you’ve chosen a different title or are still considering your options, I’d appreciate a quick update so I can ensure our records are accurate.
Also, please don’t forget to check with your bookstore about any Equitable or Inclusive Access programs that may be available. These programs can provide your students with more affordable digital access to their course materials.
I am sincerely grateful for your attention to this matter; I look forward to the opportunity to hear from you at your earliest convenience.
Cordially,
**
Sales Support Assistant
Higher Education Sales, Americas
**
On one hand, hey, I’m thrilled that they’re promoting our book. On the other hand, it seems like they’re kinda flying blind if they’re sending this promotional message to the author of the book.
How to avoid the “clean data, clean model” trap when teaching statistics and data science
Parthsarthi Joshi writes:
There is a stark problem that I have observed with how data science is taught in most books and tutorials – the concepts are taught on an individual level but fail to give a complete idea. The so-called “real-world” projects are carefully orchestrated like an “act” already setup by the instructor. The students are not given the taste of the real world – where data is messy, models do not continuously improve, and assumptions are almost always broken. We teach them the concepts of statistics and ML but not the art of data science, which includes curiosity and hit-and-trial experiments.
I am writing a one-of-its-kind book on Data Science that would be completely based on projects. The data will be messy, not everything we do will improve the performance. The aim is to give a feel of uncertainty and challenges that promotes learning from failure and develops an analytical bent of mind.
I agree with Joshi’s concerns, and it’s something that’s bothered me for a long time. Textbooks and research articles are full of examples that work, they’re full of estimates that are a bit more than 2 standard errors from zero, everything’s a bit too clean.
I’ve tried to push against this in my own teaching and writing. Here are some examples:
1. Decision analysis textbooks often use examples that are trivial and artificial (a college student is trying to choose an apartment or dorm room, balancing the factors of cost, amenities, and commute time) or important but vague (a company is deciding where or whether to build a new plant, it’s an interesting problem of decision making under uncertainty but the details are not given). A bit reason I wanted to do the full decision analysis with Phil for the home radon problem was that I wanted a real problem, with real data, going from beginning to end.
2. A few years ago there was a heart-stent study reported in the news. The experiment yielded results that were not statistically significant (the p-value was 0.20) and it was inappropriately reported as a null effect. We looked at the paper carefully, and a more reasonable analysis gave a p-value of 0.09–still not reaching conventional levels of significance. We wrote this up, and it’s a good example where the result is not clean. Again, something you don’t usually see much discussion of in textbooks, applied research papers, or articles on research methods.
3. Our book Active Statistics is full of stories with real-world complexity.
4. Appendix A of Regression and Other Stories, on computing in R, has examples where you have to download and clean the data. One problem with a lot of computer-friendly textbooks is that they’ll have clean data already loaded into packages so you can just click and do the analysis. I like there to be a bit of a struggle, to give a sense of what data analysis really feels like.
We try also to convey a sense of real-world complexity in the worked examples and homework problems in our textbooks.
The flip side of this is that, when you’re using live problems in your teaching, it’s good to come to class prepared! It’s fine to show students the steps of data cleaning and the frustration of messy data and models that don’t fit–but it’s best for you, the teacher, to encounter and resolve those problems on your own first, so that you don’t confuse the students even further. In class it’s good to have a sense of what’s coming next, so that when something doesn’t work, you’re not surprised and you can contextualize it for the students right away.
Stand-up comedy—like teaching, and book writing—requires “a collaboration with the audience.”
In our bathroom we have this book, “And here’s the kicker: Conversations with 21 top humor writers on their craft,” edited by Mike Sacks. It’s well suited for the throne, as you can dip in and read as little or as much as you want.
The gimmick is that you’d buy the book if you wanted to enter the comedy field. Which doesn’t seem very likely, but, then again, who reads books either? So maybe I’m one of the few people reading this one for entertainment value alone.
One of the interviews is with Woody Allen collaborator Marshall Brickman, coauthor of the 1970s classics Sleeper, Annie Hall, and Manhattan. Lots of interesting stuff in this interview, including this:
Marshall Brickman: [Woody] found a whole new area of insight: relationships of a certain kind, psychoanalysis, and the creation of the so-called loser—mostly with women. To some extent, the lower-with-women character was someone Bob Hope would play, but in a much more general and mainstream way. Woody’s character was more ethnically and culturally specific.
Mike Sacks: It takes true genius to develop a comic character like that.
Brickman: It does, but it also requires a collaboration with the audience. It’s the only way you can do it. You have to get out there and do a variety of material. Over times, certain things, statistically, will continue to work, and other things will drop away, and the audience will tell you what seems correct for you—for what you project onstage as a personality.
Sacks: But even with that said, you can work for twenty years and never connect with the audience half as much as Woody Allen.
Brickman: That’s right. That’s the genius. Creating something that somehow resonates with an audience that strongly.
If only Allen had died in 1986 (after releasing Hannah and Her Sisters) or maybe 1992 (after Husbands and Wives), just think how high his reputation would be. Even setting aside his personal life, releasing a series of flawed projects takes a toll. Not everyone’s a Scorsese or Spielberg who can keep it up into their old age. If Woody had gone in 1986, we’d still be speculating about the amazing work he didn’t have the opportunity to do.
But the thing that really interested me in the above-quoted interview is what Brickman said about collaboration with the audience.
This is true of teaching too! Teaching a class, even writing a book, is a collaboration with the audience. A book doesn’t read itself. With books, the challenge is that the audience doesn’t see it until it comes out. To continue the analogy, a live class is like stand-up; a book is like a movie.
When writing books, we always have the audience in my mind. The trouble is that the book might never reach the audience. That’s what I think has happened with Active Statistics. I’d like to rearrange it and rerelease it as a book called Statistics Stories. Or maybe, Important Concepts of Statistics in Story Form. Also I wish we’d given Regression and Other Stories the more serious and descriptive title, Applied Regression and Causal Inference. Maybe we can rerelease it under that title.
Anyway, yeah, “collaboration with the audience”: definitely.
Why quantitative understanding of effect sizes matters, even if all you care about is the presence of the effect
In reaction to my article with Andy King proposing post-publication review, Dan “Fast and Frugal” Goldstein writes:
Your process limits information search, computation, and time so it seems fast and frugal to me. Happy you still associate me with that term. It was something I coined as a grad student.
In your proposal, only hit papers get audited. It reminds me a bit of Mel Brooks’ The Producers in which the protagonists use the logic “who would audit a flop?” and stay under the radar by intentionally producing a bad show. Fraudsters have likely attempted to make their work seem worthy of publication while ensuring it doesn’t attract too much attention. Just like in The Producers, though, this sometimes backfires.
I don’t know about that! My impression with fraudsters is that they think that fraud is normal science, perhaps out of some mixture of bad education in research methods, a view that “everybody does it,” and a general lack of understanding of how non-cheaters (like you and me!) think. Think of people like Wansink who gave general advice to to the world on how to p-hack, or Gino and Ariely, who published papers on dishonesty, or Mary Rosh, who surely believes that whatever shady statistical manipulations she does are nothing compared to the dastardly deeds done by the Democrats.
I’m sure there’s tons of below-the-radar cheating and bad science that we don’t hear about, but a fair number of prominent science fraudsters seem to enjoy the limelight. One reason for this seemingly self-sabotaging behavior, I think, is that cheating enabled these people to attain great professional success for years. They had no reason to think the juice would stop flowing.
To return to my proposal with Andy King: I think it’s ok that only the hit papers get audited. Bad papers that get no intention aren’t doing much damage, right?
Goldstein adds:
By the way, I was just having a conversation about your sensing that something was amiss with the LaCour study. For years now I have used this quote of yours in a talk I give about putting numbers into perspective. I argue that it’s really important that people learn how to put numbers into perspective because if they don’t, they won’t notice that something is unusual and worthy of a deeper audit. You somehow sensed something was up with the Lacour result. You didn’t think it was fraud yet but you knew it was strange because you know how to put such differences into perspective:
A difference of 0.8 on a five-point scale . . . wow! You rarely see this sort of thing. Just do the math. On a 1-5 scale, the maximum theoretically possible change would be 4. But, considering that lots of people are already at “4” or “5” on the scale, it’s hard to imagine an average change of more than 2. And that would be massive. So we’re talking about a causal effect that’s a full 40% of what is pretty much the maximum change imaginable. Wow, indeed. And, judging by the small standard errors (again, see the graphs above), these effects are real, not obtained by capitalizing on chance or the statistical significance filter or anything like that.
My colleagues and I recently wrote a paper on this general topic of average effect sizes. It’s our contention that people generally are way too optimistic about possible effect sizes, in large part because they don’t think about variation. If you ask someone to hypothesize an effect size, you’ll typically get a guess of the largest effect that might occur.
But what if you don’t really care about effect size–you just want to know about the effect?
For example, maybe you don’t believe that women during certain times of the month are three times more likely to wear red or pink shirts, but you are interested in some sort of evolutionary psychology theory of sexual display. In that case, why should the effect size matter? Why care that a study reported an estimate that was ridiculously implausible?
I have two to this questions, and thus two reasons why effect size is important even for problems where you don’t directly care about effect sizes:
1. Effect sizes vary. An treatment that has an effect (that is, a true effect, not just an estimated effect) of 0.1 for one group of people in one setting could have an effect of -0.2 in some other scenario. A treatment effect in an experiment is the sum of all sorts of things, positive and negative, and there’s no logical reason to think the sign of the effect will be preserved. Effect size matters. The issue is not just that a smaller and more realistic effect size is less important; it’s also that smaller effects can be more easily produced by other factors, and this reduces the generality of any claims, even if the experiment at hand was done well.
2. Experiments produce standard errors as well as estimates. If the standard error from a study is large compared to any realistic effect size, then the study contains very little information. Effect size is important in understanding the informativeness of an experiment, and to do this right you need to have some sense of what the true effect size could be. You can’t just use a point estimate from the study itself, as this estimate will inherently be too noisy to use to judge the information in the study. As I wrote in this note for the Annals of Surgery, Post-hoc power using observed estimate of effect size is too noisy to be useful.
A tool for learning about Fourier transforms
Eric Novik came by my talk the other day and we were chatting about a number of things, including how much we forget as the years go by. I remarked that I used to be very comfortable with Fourier analysis and was able to use it as a research tool—see section 2.2 of my Ph.D. thesis, and it also came up in my research leading to R-hat (although it didn’t make it into the writeup)—but at this point I only understand Fourier analysis on a conceptual level. It’s not one of these things that stuck with me.
In response, Eric pointed to this app that he created (with chatbot assistance) to help him my understand some things about Fourier series. Maybe it will be useful to some of you too. The source code is here.
Podcast coming on Bayesian workflow! With a contest!
Alexandre Andorra will be interviewing Aki, Richard, and me on the topic of our just-released book, Bayesian Workflow.
And there’s a contest! Here’s Alexandre:
🥁One listener will get to bring their real-world workflow problem onto the recording for the three of them to work through live!
🎟️ How to enter:
1. Like, comment or repost this announcement post so other practitioners see it
2. Review the show on Apple Podcasts (https://lnkd.in/gbTagKAS) or Spotify (https://lnkd.in/guYMdMUv), and upload a screenshot here: https://lnkd.in/dPpuSKf5
3. Submit your problem (100-200 words) on the form🎁 Prizes:
1. Grand prize: your problem worked through live by Gelman, Vehtari, and McElreath during the recording + a signed copy of Bayesian Workflow
2. The two runners-up will get signed copies
3. Five LBS Patreon subscribers (random draw) will get ebook copies!Contest closes June 10!
If you’ve ever wanted to come on the show and ask Andrew, Aki or Richard “okay but what would you actually do here?” — this is your shot!
And here’s the Google form for the contest.
I’ve been on Alexandre’s podcast twice before:
• Modeling the US Presidential Elections (with Merlin Heidemanns), 1 Nov 2020
• Regression and Other Stories (with Jennifer Hill and Aki Vehtari), 30 Jul 2020
It was fun both times. I assume this new one will be too.
Full day Stan tutorial at Modern Modeling Methods (M3) this summer in New York (22 June 2026)
This post is from Bob
Mitzi Morris and Bob Carpenter, two of Stan’s developers, will be presenting a tutorial on Stan and Bayesian data analysis aimed at psychometricians this summer.
- Modern Modeling Methods Conference (M3), Fordham University Lincoln Center Campus, Manhattan, June 22–24
- Program here
Abstract
This workshop is a full day, hands-on introduction to Bayesian modeling and statistical inference using the probabilistic programming language Stan.
The course will be organized around the key properties of Bayesian statistical modeling for science, including the nature of uncertainty, modeling a generative process through a data generating distribution, modeling existing knowledge through a prior, and pushing uncertainty through inference. As we do this, we will show how Stan can be used to both code the models and perform statistical inference for quantities of interest, be they retrospective parameter estimates or prospective predictions or forecasts. We will concentrate on full Bayesian posterior inference, including a discussion of calibration, model checking for both prior and posterior inference, and model comparison with cross-validation. We will spend some time showing how some structural equation models (SEM) can be translated directly to Stan and will also introduce psychological models for educational testing, crowdsourcing, rating and ranking, and real-time decision processes.
This class will require a notebook computer with a network connection (Wifi will be available in the classroom). We will use the Stan Playground, which runs Stan in the browser, which we will pre-populate with models of interest. We will probably also break into R or Python at various points to demonstrate methods not yet supported by the Playground, such as the brms regression expression language.
Andrew on psychometrics
Andrew once told me that any model you could come up with was probably invented by a psychometrician 50 years ago (make that 60—he said it at least 10 years ago). I have evidence that he’s right form the project that drew me into Bayesian statistics—crowdsourcing. Andrew and Jennifer Hill helped me formulate a crowdsourcing model where raters give you noisy measurements of underlying categorical variables (e.g., they answer survey questions about whether a word in context is a noun, for example, to use something I was working on at the time). Turns out Phil Dawid and A.P. Skene published the same model in 1979 in one of the earlier applications of the expectation maximization (EM) algorithm and they used natural language data (drawn from medical records).
The rest of the conference
The rest of the program looks really great—it’s just the kind of applied wrestling with real data that I like.
Speaking of Jennifer Hill, she’s one of the keynote speakers at M3. Every talk of Jennifer’s I’ve attended has been great. You may know her as Andrew’s co-author on the regression books, which I cannot recommend highly enough if you’re interested in this kind of applied modeling.
Beyond the conference
You see the same kind of Bayesian modeling focus for real data at venues such as ISEC (international ecology conference to which I went to once just because I like these models and these kinds of conferences), StanCon (see you in Uppsala in August!), and GeoMed (which Mitzi attends). I’m sure there are more in other fields. I’m always disappointed that there’s almost nothing like these kinds of nitty-gritty applied papers at ISBA (Nagoya this summer) or BayesComp (somewhere next year). The conferences about Bayesian statistics or computing that I’ve been to have all been super theoretical.
Jonah’s seminar tomorrow: “Bayesian Workflow and the Software That Shapes It”
This is Leo. Jonah Gabry (Stan developer, Andrew’s collaborator, etc.) is spending the whole month of May as a visiting professor here with us at the University of Trieste in Italy. Tomorrow, May 19th, in the De Finetti room at the University of Trieste, at 9 am NYC time (GMT-4), Jonah will give the following talk:
“Bayesian Workflow and the Software That Shapes It”
based on the upcoming book: “Bayesian Workflow”.
For anyone local, you are welcome to come in person. Anyone else can join on Microsoft Teams (available here).
Expanding the Stan User’s Guide
This post is from Bob.
The Stan User’s Guide has been evolving organically along with the project. I’m writing this post for two reasons—to let you know what we’ve added lately and to encourage you to add your own chapters.
History
Initialy there was just one doc that included what is now the Reference Manual, Function sReference and CmdStan Manual. You can browse them all on the web or download pdfs from the Stan documentation web site.
Most of the topics grew up opportunistically from either things the team was interested in, new features we added to Stan, or just trying to cover the basics of the chapters of Bayesian Data Analysis and Data Analysis Using Regression and Multilevel/Hierarchical Models.
Recent additions
There have been some recent additions, including some from new faces:
- Franziska Henrich: drift-diffusion Wiener models for reaction-time modeling
- Abner Heredia Bustos: multiple imputation
- Brynjólfur Gauti Guðrúnar Jónsson: copulas
- Charles Margossian, Steve Bronder, Aki Vehtari: embedded Laplace approximation
- Me: survival models
There are also a metric ton of minor clarifications, fixes, and model examples, such as Mitzi Morris’s additional sufficient statistics optimizations and you can simply look at all the documentation pull requests.
The future is you
You can see from Franziska’s pull request, Abner’s PR, and Brynjólfur’s PR that we provide a lot of guidance and feedback. I found it super useful to write a lot of the documentation when I was learning statistics because I got feedback from Andrew Gelman, Ben Goodrich, Aki Vehtari, and Michael Betancourt, among others.
Some topics that would be nice to add
There is a lot to add to our existing discussions and you can see a lot of that in the pull request list on our documentation repository. There are also many topics that we don’t cover or don’t cover well. Off the top of my head, roughly ordered near the top according to my assessment of how impactful it would be to add.
- causal inference using Rubin’s potential outcomes framework
- Thompson sampling for reinforcement learning/bandits (sorry, Andrew!)
- penalized complexity priors and the Besag-York-Molie II spatial model (working from Mitzi’s Morris’s case study)
- Extreme value models
- RNA-seq and DNA-seq composition models at the read level and k-mer level
- Econometric models (I don’t even know what these are, but there are books)
- Stationarity-constraining parameterizations of VAR models a la Sarah Heaps
- Neural networks
- A/B testing (ideally sequential)
- spatio-temporal models along the lines of Leon Held’s survey
- non-trivial examples of ODE models like Lotka-Volterra (my case study), SEIR (Elizaveta Semenova et al.’s case study) or even better, pharmacokinetic/pharmacodynamic (multiple case studies around Torsten), soil carbon (my very early case study, but there’s been much better work since)
- Hilbert-space approximations to Gaussian processes (GPs)
- PDE approximations to GPs
- Fourier space analyses of GPs
- N-gram language models
- Plackett-Luce model for contest with more than two players (this can follow my case study for StanCon)
- …
Some of those may already be mentioned in our doc, so sorry if you’ve already written something. Also, feel free to suggest more topics in the comments.
Probability theory corner: My favorite birthday-problem story
Since we’re on the topic of the birthday problem, I wanted to share a story from my review of Dan Davies’s book, Lying for Money:
On p.124, Davies shares an amusing story of the unraveling of a scam involving counterfeit Portuguese banknotes: “While confirming them to be genuine, the inspector happened to find two notes with the same serial numbers—a genuine one had been stacked next to its twin. Once he knew what to look for, it was not too difficult to find more pairs. . . .”
The birthday problem in the wild!
P.S. I sent this story to John Cook, saying that this was the first time I think I’ve ever seen this particular problem come up in real life. John replied that birthday problems come up all the time in cryptography, e.g. hash collisions, and he pointed to this post from 2017:
Ideally, a secure hash is “indistinguishable from a random mapping.” So if a hash function has a range of size N, how many items can we send through the hash function before we can expect two items to have same hash value? By the pigeon hole principle, we know that if we hash N + 1 items, two of them are certain to have the same hash value. But it’s likely that a much smaller number of inputs will lead to a collision, two items with the same hash value.
The famous birthday problem illustrates this. . . . Variations on the birthday problem come up frequently. For example, in seeding random number generators. And importantly for this post, the birthday problem is the basis for birthday attacks against secure hash functions. . . .
I had no idea! John also points to Pollard’s rho algorithm as another real-world application of the birthday problem.
Fascinating 1981 interview with Morris Kline, author of the classic book, Mathematics: The Loss of Certainty
From this 1981 interview:
So when did the loss of certainty begin? Where did we take a wrong turn?
It began around 1800, and it began with geometry. I usually like to quote Mark Twain about this. He said that man is the only animal that has the one true religion—several of them. And that is just what happened with geometry.
The geometry that came from the Greeks is usually called Euclidean geometry, after Euclid. But suddenly at the beginning of the 19th century other geometries were developed—non-Euclidean geometries. Who gets the credit for this is sometimes disputed among historians, but I would say Carl Friedrich Gauss. He was the man who said flatly that we can no longer be sure that Euclidean geometry describes the physical world correctly. The various geometries conflict, although one of them, according to thousands of years of tradition, should describe the truth. You can see the problem.
Can you give me an example of an alternative geometry?
Well, one can cite as an example the theorem of Euclidean geometry that the sum of the angles of a triangle is one hundred eighty degrees. In one of the non-Euclidean geometries, called hyperbolic geometry, the sum is less than one hundred eighty degrees; in another, called double-elliptic non-Euclidean geometry, the sum is always larger than one hundred eighty degrees. Yet all of these geometries are equally accurate insofar as man can measure the sums of angles of triangles.
I actually disagree with Kline on that point! Draw a (virtual) triangle connecting the North Pole to two points on the Equator that are 90 degrees apart in latitude, and each angle of that triangle will be 90 degrees. 90 + 90 + 90 = 270: we can measure that.
The interview continues:
What did the mathematicians do when the bottom dropped out of geometry so to speak?
Many mathematicians tried to rescue and maintain as truths the portion of mathematics built on arithmetic, which by 1850 was far more extensive and vital for science than the several geometries. Unfortunately, other shattering events were to follow. Arithmetic and algebra were the next to go by the board.
The best example of this I could give in a semi-popular book was the creation of what are called quaternions, in 1843, by the great mathematical physicist William Rowan Hamilton. Now in the algebra of quaternions, a kind of number known as a hyper-number, multiplication is not commutative. In other words, if I were talking quaternions, I could not say that three times four is the same as four times three. Other strange algebras were created, and it made people start to worry about the laws of ordinary arithmetic. (The one I just stated is known as the commutative law of multiplication). And if we can have perfectly good algebras in which the old familiar laws don’t work, then how do we know they work in the case of the real numbers? That’s where a mathematician named Hermann von Helmholtz stepped in and told us we don’t know it at all. They work in some situations, but not in all.
Are there any elementary examples of these sorts of algebras, where 2 + 2 = 6, or where 5 x 7 = 35, but 7 x 5 is only 34?
I can think of several. Take a quart of water at 40 degrees and mix it with another quart of water at 50 degrees. Do you get two quarts at 90 degrees? You do not. It’s more like 45 degrees. So you can’t just say I’m going to add 40 and 50 and automatically get 90. It depends on the physical situation.
Consider music, a simple musical tone with a unique frequency and amplitude, say one hundred cycles per second. Now suppose on top of that you impose another note at two hundred cycles per second. Do you get a note at three hundred cycles? Again you do not. It is a note of two hundred cycles, the first harmonic above the one-hundred-cycle note. It is the highest harmonic that determines the pitch—two hundred cycles. This is an important factor in the design of musical instruments.
Those are excellent examples. And recall that the laws of probability do not apply in real life (that is, quantum mechanics; see section 2 of this article).
This is interesting:
If mathematics has no underlying truth–if it is filled with contradictions and uncertainties, why does it work?
There is no definitive answer to that. It just works. The only test we have that mathematics is reliable–not certain, but reliable–is that one can apply its laws to physical problems and make predictions. If the predictions come through, then we can say that mathematics has some substantial basis, but not certainty. I think people can’t help being impressed by what mathematics achieves. Consider the problem of sending a spaceship to the moon and bringing it back. It is entirely mathematical. Of course, there is a tremendous amount of engineering involved in the production of the ship, but the entire plan for it is mathematical. We have a theory about the sun, the planets, and more distant heavenly bodies. We say that what makes them behave as they do is the force of gravity. But nobody knows whether there is such a thing as gravity. We have no physical understanding of it. The theory is mathematical–gravity is a scientific fiction.
The same could be said about electricity and magnetism, couldn’t it?
That’s exactly right. Everybody today knows what a radio is, and what a TV is, but nobody knows what a radio wave or a TV wave is. You can’t smell one or hear one or taste one. But we do have a wonderful mathematical theory developed in the nineteenth century by the mathematical physicist James Clerk Maxwell. The evidence for this wonderful theory is the performance of our radio and TV sets. So we have to accept the fact that mathematics works, or else abandon our radios and our TV sets.
I pretty much agree but I’d put it slightly differently. There are various mathematical theories that don’t work, and because of that we don’t use them to design radios and TV sets. To put it another way, it’s not quite right to say that “mathematics” works; rather, some branches of mathematics work. For example, you could think of various goofy variants of logic and probability as mathematics, but nobody’s using them to build ships. Or, for another example, physicists don’t use classical (“Boltzmann”) probability in quantum problems, because . . . it doesn’t work. The mathematics that works, that’s the stuff that works. Any bit of mathematics works until it doesn’t, which is the point where people try to push it beyond its bounds of applicability.
I enjoyed this bit:
Are most mathematicians since the loss of certainty now working on these physical problems?
No, they aren’t. Most of the mathematics created today–maybe ninety percent of it–is a waste of time. That is an opinion, but one that authorities who are far more creative and far better known share with me.
Can you give us an example of mathematics you consider a waste of time?Some problems now being considered in the theory of numbers, for example, are a waste of time. Take pairs of primes, called double primes. These are prime numbers in a sequence, eleven and thirteen, for example. No even numbers, of course, are primes. Are there an infinite number of these pairs? Are there triple primes? Endless papers are written about these subjects. Who cares?
That’s how I feel too! That said, I understand that the sorts of insights required to solve this sort of number theory problem are cognitively similar to the sorts of insights required to solve what I would consider to be interesting and important problems in mathematics and statistics. So, even though I agree on “Who cares?”, I don’t think that research in this area is “a waste of time,” any more than it’s a waste of time if you’re an athlete to do cross-training.
The interview continues:
It makes mathematics sound a lot like playing chess or bridge. Exciting, beautiful, challenging; the same words apply to all three kinds of activity.
That’s right. I’m glad you suggested it because it makes the point sharper. People enjoy playing chess. Some people even devote their lives to it. But no matter how ingenious a man is at playing chess or bridge, it isn’t going to change this world one iota. Now mathematicians may probe deeper problems, but it is the same thing.
Again, I kinda feel that Kline is missing the point. For one thing, the effort spent to build programs that can win at chess and go has led to general improvements in machine learning and AI. For better or worse, the study of chess has changed the world, and by more than one iota.
Overall, I’m a big fan of Kline and I like a lot of what’s in that interview, which is one reason it’s interesting to see where we disagree.