Regular readers know that I’ve been screaming for years about the problems with binary thinking, for example here and here. Or, for a applied example from medical research, here.
This comes up in statistics all the time, when “statistically significant” results are treated as representing enduring truths (I’m looking at you, beauty-and-sex-ratio, ages-ending-in-9, ovulation-and-voting, subliminal-smiley-faces, ESP, and all the rest), and non-statistically significant comparisons are treated as no effect.
But today I came across another example which made me think that the flaws of binary reasoning arise in slightly more complicated settings too, when people take a probabilistic argument and reduce it to a false syllogism.
I’ll give the example and then discuss the more general problem.
The example
It comes from a post by Tyler Cowen, not something he’s saying himself, exactly, but I think by posting it he’s endorsing the argument, or maybe it’s a bit of mood affiliation, as Cowen might say. Anyway, here it is:
My first reaction upon hearing that boosters were rejected was to ask the same thing: would these same “experts” say that, because the vaccines are still effective without boosters, vaccinated persons don’t need to wear masks and can resume normal life? Of course not. They use the criterion “prevents hospitalization” for evaluating boosters (2a) but switch back to “prevents infection” when the question is masks and other restrictions. What about those that are willing to accept the tiny risk of side effects to prevent infection so that they can get back to fully normal life? The Science (TM) tells us that one can’t transmit the virus if one is never infected to begin with.
Also, one of the No votes on boosters said that he feared approval would effectively turn boosters into a mandate and change the definition of fully vaccinated. So, it appears that the overzealousness to demand vaccine mandates has actually contributed to fewer people getting access to (booster) vaccines, thus paradoxically contributing to spread. A vivid illustration of the problem with, “That which is not mandatory should be prohibited.”
Why the reasoning is wrong
I’m not saying the above passage is wrong on the merits. Drug approval involves lots of tricky questions, and it’s worth wrestling with these difficulties. Indeed, if the passage were just plain stupid, there’d be no point in discussing it. What I’m getting to is that I think the author of the passage is making a sort of logical error arising from inappropriate discretization of choices and outcomes.
So here’s my concern with the above passage: It takes decisions about mask wearing and social distancing and reduces them to the binary of:
(i) Wearing masks, or
(ii) Resuming normal life.
But it’s not just (i) or (ii). Last semester I started teaching in person and meeting in person, and I’m loving it and loving it. So that’s my bias, or my personal preference. But we were also wearing masks in the classroom (except I was allowed to not wear a mask when teaching and more than 6 feet away from any student, so sometimes I pulled off the mask to take a drag of air) and in the office (except, again, for the occasional socially-distanced drag). Now you might say that we shouldn’t have bothered with masks (or maybe that masks shouldn’t have been required)—but there were many people on campus who didn’t think we should’ve be teaching in person at all!, and I understand the university had to respect that perspective. This gets to the cost-benefit issue: the cost of wearing masks is just not so high. But the cost of getting rid of full in-person classes is high, not in terms of health but in terms of education.
It seems to me that the passage quoted above is a sort of syllogism, nailing the so-called “experts” for a logical inconsistency—but the apparent illogic is only arising because the decision is being taken as binary.
The other discretization in the above quote, and this is important too, is in its treating of “these same ‘experts'” as a unified mass. I think the author of the quote is a bit too sure about what these “experts” (I guess it’s some FDA employees, or people on an FDA panel, or something like that?) would say about various other policy questions. You can make lots of errors in reasoning if you condition on facts that you’re not sure of. To put it in statistical terms, it might be that you have good reason to believe each of the statements A, B, C, D, and E—but if you condition on all five of these plausible statements, you might well lead yourself into a strong and unreasonable inference.
The general problem
Again, I can understand the frustration being expressed by the author of the above quote, and so don’t take this post as a defense of any particular policy regarding vaccine boosters, or as a defense of any particular mask policy. My point is a more general one, that the logical form of the syllogism is seductive, but it breaks down under uncertainty.
There are lots of examples. For example: “A is correlated with B” and “B is correlated with C” does not necessarily imply the statement, “A is correlated with C.” Confusion on this can lead to all sorts of problems. This syllogism thing is different, but it seems to be me to be of the same family of fallacy. I’d be interested in reading a more systematic treatment of the topic: deterministic arguments fall apart in the presence of uncertainty. Another example is the “What does not kill my statistical significance makes it stronger” fallacy.
It’s an interesting problem where the use of logic can pull you into fallacious reasoning—but, at the same time, the apparent logic makes the (erroneous) argument more compelling, which in turn can lead you to think that people on the other sides are knaves or fools, which then makes you less likely to listen to explanations of why your reasoning is wrong, which . . . It’s a vicious circle out there!
There’s also a political science angle here. Vaccine denial is annoying and it’s sad, and policymakers need to account for it. Politically-minded promotion of vaccine denial is even more horrible, and policymakers need to account for that too. A bit less annoying are people who are so scared of covid risks that they resist return to school. Policymakers need to account for them too! Wearing masks indoors and following testing protocols can be a way to satisfy that group.
To return to the “vaccinated persons don’t need to wear masks and can resume normal life” issue: I’d rather not wear a mask at all, but if I can teach classes and do meetings in person (and, in my non-work life, if I can travel and see people and go to the store etc.), than that’s close enough to “resuming normal life.” I mean, sure, I’d rather be able to ride the train without a mask—that’s not nothing—but “normal life” is a continuum.
Andrew wrote:
“This gets to the cost-benefit issue: the cost of wearing masks is just not so high. But the cost of getting rid of full in-person classes is high, not in terms of health but in terms of education.”
The debate about masks has been going on since Covid 19 hoved into view. I propose a more recent controversy, math textbooks and Florida:
“This gets to the cost-benefit issue: the cost of teaching math properly in Florida is just not so high. But the cost of getting rid of uncomfortable examples is high, not in terms of health but in terms of education.”
I’ll take the opportunity to hijack this thread a bit. The example from the Florida case that interests me is the one that relied on the Implicit Association Test. The book that was rejected had a bar chart that students needed to work with that involved age distributions of results from the IAT. The text refers to the test as measuring “levels of racial prejudice.” In my opinion the textbook that was rejected deserved to be rejected – but not for the reasons used in Florida. In fact, the reason for rejecting are atrocious – and the book is atrocious for other reasons entirely.
First, the rejection is part of the growing trend (particularly evident in Florida) that seeks to remove anything having to do with race (or gender or…..) from school curricula. Should math education be devoid of social content? I would say no; in fact, I think using controversial and provocative content is precisely what more textbooks – including math texts – need to do.
But the problem with this particular example is that it declares the test as measuring racial prejudice rather than taking the opportunity to discuss measurement (which, in the case of the IAT, is controversial), experimental conditions, disparate research results, etc. In other words, this was an opportunity to engage students in a living example of issues with measurement and interpretation of results. Instead, the book used it as an example of reading bar charts and estimating algebraic equations from the chart. Those goals are fine in themselves, but if approached via an emotional subject such as racial prejudice, deserve to be approached not as facts, but as theories. Theories open to testing and evidence.
So, in my mind the book was rejected for the worst of reasons, but also provides a terrible example of what the content could have been. Thanks for the opportunity for this digression that allowed me to get this off my chest.
There’s also the false dichotomy that in person is better than remote. Plenty of students thrived on remote learning, some really don’t want to go back for various reasons.
The world is complex and variable, everything is a distribution, this is the single most important idea we could be teaching in statistics. Instead we teach t tests for differences in means…
That’s a good point Daniel, there’s a lot of heterogeneity, and some people have thrived under remote (I don’t know many students who did, but certainly a lot of workers did).
Still, I feel it’s a bad idea for some students to just always remote into a normal class. The problem isn’t on the student’s end, but with the instructor who has to accommodate separate groups. In-class activities become impractical, moving around the room too much can’t be done without camera tracking, tests become a pain, etc.
I hear some K-12 schools are doing remote-only classes for students and teachers who want to remain remote. That seems more doable. Although I feel a goal of school at that level is to build social skills, and that seems hard to pull off remote.
I recently ended up with a convenience sample of students in a university biology class who opted not to participate in a course field trip to a remote, beautiful research station (think whales, sunsets, and Jacques Cousteau). All students paid an extra course fee for this field trip, there was a small grade penalty for not attending, and there was no on-line equivalent activity or assignment. In spite of those incentives & costs, many students chose not to go. Many members of that group also skipped labs and viewed lectures as video recordings. By the end of the semester, that group had much lower mean and median scores on all parts of the course compared to students who did go on the field trip. The difference is consistent with several post-hoc interpretations, but one of them is that many students who prefer to avoid in-person class activities are misleading themselves about the effectiveness of remote learning for themselves (admittedly some other students prefer remote and do very well in that mode).
Your conclusion, standing on its own, is a reasonable one. And I agree with it 100%. People who prefer online coursework almost certainly overestimate its effectiveness, generally. But your data contradict the conclusion that that was a major factor in this *particular* case. It looks to me like something completely different was going on for your sample, convenience or not.
Consider: you’ve described an in-person activity that, unlike most coursework, was hugely and multiply incentivized. For these students to choose not to go on the trip they had to, as you posit, dismiss its potential academic benefits. But then they also had to not be tempted by a truly unique and wonderful experience. And they also had to not care about wasting the money they’d already paid.
The failure of any of those incentives to work implies that they had huge counterincentives not to go. Which would mean there was something–probably many things–going on with members of this subgroup. Things like, maybe they were sick or recovering. Maybe they were caring for someone in a high-risk group. Maybe they had personal commitments or pressures that make it difficult to meet in person, much less go on field trips. In which case, contrary to your conclusion, their motivations may have been heterogeneous and their decision to opt out may have been not only rational but genuinely wise.
Again, I agree with your opinion, but I thought it worthwhile, on a post about oversimplification, motivated reasoning, and logical fallacies, to question the empirical rationale you supply for it.
Oops, forgot to make the point that any of those alternative explanations could have directly caused the bad grades, regardless of the effectiveness of online learning or their evaluation of it, making the correlation between missing the trip and bad grades spurious.
Exactly what Mike said…
A post of mine from a few weeks ago echoing that: https://eighteenthelephant.com/2022/03/27/the-perils-of-online-learning-in-one-graph/
It’s a challenging issue, of course, since there’s wide variety among students, and since students should be allowed to fail…
You do a great job of detailing caveats/weaknesses with your analysis, and you note that you can’t control for whether weaker students took online options, in which case the causal arrow may be reversed. But it seems to me you could easily check whether grades went down overall from the previous year. If they didn’t, or if they didn’t by much, or if they did but the shape of the distribution stayed the same, that would support the idea that online attendance didn’t cause worse performance, right? In other words, just plot histograms for last year and for this year, and if online attendance causes poorer performance, the most recent year’s performance should either be multimodal or very right-skewed. That kind of distributional difference on its own wouldn’t imply a negative effect, but a negative effect (all else being equal) would imply that kind of difference.
Mike, Raghu:
I agree, and I feel the same thing about the trend in the past decade or so toward college classes meeting just once a week (instead of the previous standard of two times a week, or the standard before that of two or three times a week). Students love once a week because then it’s easier to schedule classes, also they can try to schedule all their courses in one or two days and then not have to go into school most of the week. Faculty love once a week cos it’s less work. Administrators love once a week because it makes things easier to schedule. But I think it’s bad all around. In the short term lots of people like to work less, but all this goes against the larger goal of learning.
I thought we were more interested in variation than average. Some students can learn more effectively once per week than twice and some the reverse (the easiest case is to compare full time undergrads with working graduate students). There is extensive self-selection when students have a choice of in-person vs. online. Even worse is when they can choose within the same class (as far as I am aware, “hybrid” approaches that involve some students in person and others at a distance in the same course has been almost a universal failure). Online has so many different varieties that I think it is counterproductive to talk about it as if it is one distinct model. I still believe that some material is better pre-recorded, some better in person, and some better online – and it varies with different audiences. It seems foolish to me to declare that there is one best model for everyone.
Dale:
I think it’s good to have a range of options. That said, I’d like to have more of these options be classes that meet multiple times a week, and fewer of these options be classes that meet once a week. I’m not happy that the trend has gone in the other direction, and I think that teachers, students, and administrators are mistakenly over-valuing short-term convenience.
Also, yes, I agree wholeheartedly on the value of pre-recorded material. That’s why I assign books and articles for students to read before class: books and articles are pre-recorded material that each students can read at his or her own pace.
> books and articles are pre-recorded material that each student can read at his or her own pace.
+1.
Raghu’s example is super interesting but slightly different from mine. Going to Raghu’s lectures in person could potentially make a big difference in learning and performance in that physics course. But going or not going on the field trip for my biology course could make only a very small difference in the actual learning achieved by my students. Instead, it’s an index of engagement and willingness to participate in in-person activities. The puzzle for me is why a large proportion of my students actively choose to be disengaged.
Mike:
I can’t say for sure, but I think that for a lot of people, asynchronous text communication is just easier and less stressful than in-person social interaction. For example, here we are typing into our computers on a blog and communicating with people we’ve never met, rather than going outside and chatting with the neighbors. It’s my impression that students learn a lot more when they’re in the room with other students and the teacher, but I can see that, in the short term, it’s easier for a student to just stay in his or her room and participate remotely while simultaneously reading email and surfing the web.
Remote watching an in-person class is likely a garbage way to go. Designing a class for remote learning, is a totally different thing.
Effective remote learning (i think) should be asynchronous most of the time, with synchronous “check ins” and a discussion board for students to talk with each other. I think of all the things people have learned in say the Stan forum, or the Julia discourse, or OpenWrt discourse… by asking questions and getting answers. Combine something like that with video recorded lectures, written materials, and the ability to connect to a person 1-1 for brief sessions to ask questions throughout the week, and you’d get an effective learning option I think.
You also have to remember to compare it to the realistic alternative. For example for many people maybe the alternative isn’t “in classroom learning” but rather “working in a shipping fulfillment warehouse and not taking any courses at all” or something.
That has been my experience and that is the way I try to teach. I do use a bit more synchronous interaction, since students do struggle with analyzing real data sets and they learn a lot from each other – but the material I know ahead of time I want to cover, is all prerecorded.
In addition to your suggestion to compare it to the realistic (for some) alternative of no courses at all, there is another realistic comparison to make. Having spend years standing in front of a class (and enjoying it much of the time), there is the question of how engaged the students that are physically present really are. Unless you continually call on them, the belief that they are paying attention may be an illusion. Much of the time they are thinking about things other than what we’d like to believe. Conjecture on my part, but based on experience.
Dale,
When I was in graduate school (decades ago but more recent than my undergrad experience) I was a fairly mature student committed to gaining every last bit of knowledge I could from the experience. What I found was, depending on the course and the instructor, I might not be fully focused on the exact material being presented at every single moment. But if my mind drifted away from that precise content I was still 100% fully engaged with the larger subject being taught in that class.
I doubt very much I’d be able to recreate that feeling of immersion in a topic while sitting at home reviewing video snippets or watching a Zoom session of a lecture. I’m much more conditioned to interaction being fragmentary, intermittent or (in many cases) just a waste of time. Not sure I could ever replace that with a feeling of being fully “present” as I could when physically sitting in a classroom.
Which isn’t to say some sort of remote learning that would work for me is impossible to construct. Just that I’m agreeing with the idea that whatever form effective remote learning (at the graduate level) might take, it won’t be hours of Zoom broadcasts of a guy standing in front of a whiteboard.
My experience as a TA for undergraduates is that most spent class time writing down everything I (or the lecturer) wrote on the board with the plan to study it (memorize it?) right before the exam. This struck me as ridiculous. Class time is time when the students should be thinking. Having scheduled time for classes gives scheduled time for thinking. Of course, the students should also be thinking when they review the material that was covered in class and doing homework. If you are mature, then you can learn at your own pace. But, scheduling time for it helps, regardless.
How about let people decide what they prefer, why does everything need to be a monoculture these days?
If some schools want to do remote learning and students will pay for that, great. If they want to be all in-person that is also great. Diversity is good, centralizing decisions into one-size-fits-all mandates is bad.
> How about let people decide what they prefer, why does everything need to
> be a monoculture these days?
Maybe the teachers are in a somewhat better position to know what is a good way to learn than are the students. Not always, but maybe more often than not. Anecdotal evidence: When I was TA-ing math to M.I.T. undergrads, we knew better how to learn math than did the students.
Teachers are people too!
Also worth noting that heterogeneous academic performance in online vs in-person learning, even in the time of COVID, may have the same explanation as heterogeneous academic performance in every other context: inequity in education.
Is the canonical syllogism “Socrates is a man; all men are mortal; therefore Socrates is mortal” really probabilistic, since it relies on an implied inductive step: “a man = all men”?
I would say that the canonical example is deterministic.
A probabilistic example could be, “Education intervention X is correlated with increased graduation rates . Increased graduation rates are correlated with higher future earnings. Therefore, education intervention X is correlated with higher future earnings”.
I believe the last statement is not necessarily true. If the correlation between graduation rates and future earnings is less than 1, than education intervention X could be correlated with some component of graduation that is independent of future earnings.
Taleb in his critiques of IQ discusses the intransitivity of correlation, which I think is related.
How about
* People having umbrellas up is correlated with it raining outside
* Raining outside is correlated with people getting wet
* Therefore people having umbrellas up is correlated with people getting wet
I am not an educator so I may be completely out to lunch here but it strikes me that the current class/lecture paradigm is likely wrong for remote learning. I believe the lecture format evolved in the medieval period when books were rare and expensive. This seems to have evolved into the classroom with chalkboard and so on with a revolutionary change to white boards it the second half of the last century With an emergency move to remote learning due to the pandemic educators seem to generally be adapting this centuries-old paradigm to a new medium. It is not clear to me that this makes sense.
A friend of mine is a professional fundraiser for non-profits. She has recently moved to a new job with a non-profit which until the pandemic had been doing most of their fundraising from door-to-door canvassing. In the forced move to telephone canvassing they simply moved their paper “system” on line. It works but the paradigm is wrong. Spreadsheets, pdfs, separately filled out receipts, and who knows what! Her screams of frustration are terrifying her cat. One fairly basic data base would improve efficiency immensely.
I have no real idea of what kind of paradigm is needed for successful remote learning but perhaps the first thing is not to call it on-line learning. I do not see that there is any need that it be on-line[1]. Lectures may become a thing of the past with very different teacher/student interactions coming to the fore.
1. It most likely will be but the University of London was running successful degree programmes around the world long before there was an internet.
>I’d be interested in reading a more systematic treatment of the topic: deterministic arguments fall apart in the presence of uncertainty
Me too! I think of this with respect to making assertions in research papers. It may still be possible to make statements about what was learned even if one bans all dichotomous statements, but I don’t think it’s trivial the way is sometimes suggested by proponents of communication reforms. There are many pitfalls I guess in how we express thoughts.
Yet another flaw in his piece is using one person’s statement about factors influencing his decision to extrapolate to a reason for a 16-2 defeat of the booster recommendation.
The quote reads as if Cowen is using a convoluted criticism of “experts” and supporters of mandates to assert intellectual superiority or just stir up interest.