The students in my class on applied regression and causal inference are required to contribute to a shared online document. Before each class, every student needs to enter something in the document, either a question on the reading/homework or a response to some other student’s question. Then in class we discuss some of the questions that came up.
This can work really well! For example, from today’s class:
Student #1: I am a bit confused on the formula from figure 6.4. The initial equation is y = 30 + 0.54x, but I am a bit confused on how we derive to the equivalent equation y = 63.9 + 0.54(x − 62.5)
Student #2: If you multiply it out: y = 63.9 + 0.54(x − 62.5) = 63.9 + 0.54x − 33.75 ≈ 30 + 0.54x. So you have the same equation in the end. But it’s transformed because an intercept where x = 0 doesn’t make a lot of practical sense in some cases (like height, where no one can be 0 inches tall). So we write that equation instead, centering it around the mean, to give more meaning to the intercept.
Sometimes lots of students ask the same or similar questions, and we definitely want to discuss live. For example:
Student #3: How exactly is the residual standard deviation calculated, and what does it mean conceptually? How can we understand it in comparison to standard error and standard deviation?
Student #4: I also had a question about this. I find the terms “standard error,” “standard deviation,” “standard deviation of the residual,” and “standard error of the true parameter values” to at times be used interchangeably, which can be very confusing. For instance, when I read the instructions for problem 1a., I assumed we were being asked to take the standard error, meaning the standard deviation divided by the square of n. Yet, in the book it makes clear we should use the sigma score produced by the stan_glm model. Is there a way for us to remember which concept is being invoked and when going forward, or is this something we just have to discern through context and our intuition?
Student #5: Yeah, I have the same questions here. It seems that the residual standard deviation is to describe the standard deviation of the error terms in the function.
Student #6: I also have this question : is uncertainty and standard error the same thing? And is MD_SD equal to the standard error that is sd/squared(n)? And using lm, is the uncertainty printed the standard deviation or error?
So I talked about standard deviation and standard error in class. It didn’t go well. I said how the standard error is the estimated standard deviation of a parameter estimate, I wrote some notation on the board, and I did some explaining. Nothing I said was wrong, but I don’t think it helped.
And, no kidding it didn’t help! Of course my explanation didn’t help. This was a topic that the students were already confused about. More talkety talk from me is not gonna solve that problem.
Here’s what I think I should’ve done. Instead of giving this definition and explanation, I should’ve just said that the definition and explanation are in the book, and I should’ve used the class time to go over an example. For example, just a few minutes earlier we’d discussed in class this fitted model predicting the incumbent candidate for president’s two-party vote share given percentage growth in per capita income in the year preceding the election, using all the elections after World War 2:
1948-2020:
Median MAD_SD
(Intercept) 46.7 1.4
growth 2.8 0.6
Auxiliary parameter(s):
Median MAD_SD
sigma 3.7 0.7
So I could’ve just gone through each of the sd’s in the display, along with sigma, and asked the students in pairs to explain to each other the meaning of each of these. Then the next thing would be to ask what would happen to these numbers if N were multiplied by 2, or by 4.
At this point, if students wanted to ask their questions on standard deviation and standard error, we could do it in the context of this example that they’ve just been talking about. We could also loop back to their homework assignment.
My mistake was to “explain” rather than to demonstrate. Explanation is a dead end. Demonstration is forward-looking and allows students to participate, which is absolutely necessary if you want them to learn something.
And if we want to follow up, we can go through the other fitted models we discussed, breaking the data into two parts:
1948-1988:
Median MAD_SD
(Intercept) 44.8 2.7
growth 3.5 1.0
Auxiliary parameter(s):
Median MAD_SD
sigma 4.5 1.2
1992-2020:
Median MAD_SD
(Intercept) 48.4 1.5
growth 1.6 1.1
Auxiliary parameter(s):
Median MAD_SD
sigma 2.8 0.8
I think this alternative way of teaching, using examples rather than explanation, would’ve worked better for three reasons:
1. The explanation’s still there in the book for students to read when they’re ready.
2. It should be easier to follow the idea in the context of a familiar example than in abstract framing in terms of estimates and sampling distributions.
3. Even if they still don’t fully understand the concept, in the meantime they may have learned a useful related skill which will move them forward.
I like the idea of encouraging students to share things they don’t understand. I think often students are wary of sharing that, and maybe this gets even worse when there’s more they’re “supposed to know”, as with graduate students.
Obviously you’ve made them anonymous here, but are they anonymous to you and each other or not? I could see either way working well.
I always told my students, “If there’s something that you don’t understand, ask. Because if you have a question, there’s 10 other people in the room who have the same question, but are more reserved than you. So ask.”
I don’t know whether Andrew lets the students leave anonymous questions in the doc, but my experience with students is that if you let them be anonymous, you get exactly what you deserve.
Dean, Robin:
The students are not anonymous in the shared document. Indeed, they’re required to participate as part of the course. I just redacted the names when copying the comments into the above post.
These are always hard distinctions of very similar terms for students. In a lot of ways I wish the term standard error was never created because it makes it seem like something that is not a standard deviation and something that has to do with error. In a lot of undergraduate books I feel like there are sleights of hand that ultimately make it harder for students to understand. That said, having students do simulations has really changed things for me.
Reverse engineering as a didactic tool is sorely underused everywhere. Whether it’s in a higher-level programming language, Excel, or even by hand, it’s a tool every student should have some experience with.
That all aligns nicely with educational psychology research (importance of familiar, concrete examples over abstract definitions). I would also point to concreteness fading’s potential for statistics education (concrete -> pictorial/schematic -> abstract sequence of content).
Visualization of the overall map of concepts and procedures is also key on one surface (slides/board). Can you map out those definitions/contexts or examples/visual representations? This kind of thing requires more effort but should enlighten students.
A good review of educational research is:
Fiorella & Mayer, 2015 (Generative Learning Strategies)
https://link.springer.com/article/10.1007/s10648-015-9348-9
I wanted to make a remark concerning using examples, and say that that may work for some students but not others.
I browsed through the paper by Fiorella & Mayer to see if I could find a way described there that resonates with my personal experience of learning at school and university, and I think the way of “learning by summarizing” and “learning by self-explaining” are what might be most appropriate in my case.
As someone who is likely on the autism-spectrum (whatever that means), I have thought about this more and more in recent years. Perhaps especially with my brain, I have to often sort of “translate” what I hear or read in my own words. Perhaps this is an example of “learning by summarizing” and “learning by self-explaining”.
This may in part also explain my experiences with learning in high school (where I dropped out) and university (which I started at a later age and found small classes during my Masters to be much more appropriate and enjoyable than lectures in big halls with lots of people there).
I think for me to understand something, I have to go at it step by step, in my own way, having the opportunity to ask question with each step to fix possible unclarity, and overall regularly “translate” things using my own words. Perhaps it makes sense then that I didn’t resonate well with the large classes and the style lof teaching at high school and parts of university. If I miss step 1, I am out for everything that follows after that. I think that’s part of why I zoned out in high school so much, aside from not really being interested.
Just thought about the following, which might illustrate and underline the previous post.
During the summerbreak between the two years of my Research Master education, I actually attempted to learn statistics my own way by reading and working through a book called “Discovering statistics using SPSS” or something like that.
I felt I didn’t grasp the statistics well enough for me to perform my own analyses during the upcoming research project in my final year 2, even though I passed the courses in year 1. I think I learned much more by being able to choose, and find, my own reading materials (which might suit me much better), and by doing things my own way and step by step.
Ever since that experience, I have had a new appreciation for the line in the movie “Good Will Hunting” which goes something like: “(..) and two, you dropped 150 grand on a f#cking education you could’ve got for a dollar fifty in late charges at the public library.”
I would definitely call that self-explanation (a component of sensemaking). Given the flood of lecturing in classrooms, we could use opportunities to pause and think. CourseKate may be useful from a research lab at UCLA.
Andrew’s conclusion that he would have been better off using an example rather than the abstract definitions parallels my thoughts about math papers. I would often rather see an argument in a representative special case than a formal proof. The subtlety when writing is choosing a special case that really isn’t special, and convincing the editor that it’s better for the readers than the abstract argument.
I like the idea of having students read the textbook definitions on their own and covering examples in class. But how do you make sure the students are reading the textbook in the first place? I like the plan in theory but I’m nervous to test them on material I haven’t covered in class.
Maria:
I don’t know but in this case at least they were bothered enough by the topic to ask about it, so maybe they’d have some motivation to read the book again more carefully.
The real point is that whatever “explanation” I would do in class would not be as clear as the carefully-thought-out explanation that I already wrote in the book. And what is needed for those students to learn this material is not further explanation, or slower explanation, or a more vigorous explanation, but rather how to use these ideas in real examples.
I am trying this eBook this semester with my beginning students.
https://coursekata.org/
It’s interactive and so they actually get credit for reading because there are problems/questions throughout.
That’s the way it should be. Funny how far the distance is between applied educational research and what goes on in most STEM classrooms.
You need to provide examples of why any of these values matter. Eg, they lead to the discovery of DNA or something.
Discovery of DNA is a big ask.
At least in my life, I find these concepts extremely useful, and the answers to calculations are sometimes useful, not because they lead me to discoveries but because they help me avoid wild goose chases and sometimes keep me from making silly errors. I make the distinction between the concept and the calculation because the exact result of the calculation — or even a close approximation — is rarely important. Indeed, I’m not sure that in my personal experience I have even needed more than a rough approximation. But the concepts, and a rough approximation to the standard error, have been useful many times. If the average value of y this year is higher than the average value last year, do I care? Or if the average is higher at location A than location B, do I care? If the difference is consistent with noise or nearly consistent with noise then I don’t care; if it’s way too big to be noise then I might care. This has come up, in my personal experience, where y has variously represented radon concentrations in homes; corrosion rates on standard corrosion measurement coupons in data centers; monthly-average temperature-adjusted electricity usage in a building; and many more.
Just about everyone in the world should understand that as the size of your random sample increases, the within-sample variation approaches the population variability, which is not zero. And they should understand that the difference between the sample mean and the population mean decreases (in expectation), and if you have a finite population and negligible measurement error then the difference approaches zero as the sample size gets large. People who analyze data should know that for iid errors the approach to zero goes like 1/sqrt(n), should understand that if errors are correlated the approach can be much slower, and should be able to use simulation to quantify this if needed.
I think most practicing data analysis don’t need more than that, honestly. I may be wrong, but in my varied career I haven’t needed more than the simple stuff.
One reason standard error is hard is because the idea of sampling distribution is hard. And that, in part, is because the idea of random is hard. That’s why teaching research methods with statistics is so helpful, because if you have spent time focusing on simple random samples and then other probability samples in research design the idea that we don’t expect samples to have the same results becomes part of their background knowledge.
One thing I have done is to take something county level data in the US and then have each student take their own random sample of maybe 100 or 150 counties. Then, for various results, have everyone throw them into an online spreadsheet. So we are constantly thinking about sampling variation even when working on something else.
I find it odd that Bayesian statisticians still incorrectly use “the sampling distribution of the mean” to discuss standard error.
To hell with sampling distributions, that’s a Frequentist/estimator concept and totally irrelevant to Bayesian interpretation. Stop saying standard error, and just say “standard deviation of the mean parameter”. This standard deviation doesn’t mean how much variation there is in future samples etc, it means how much uncertainty there is implied by your information about the correct value of the parameter.
I strongly believe flipped classrooms is the way to go. In this approach, students study the theory as homework and engage in hands-on activities during class under the support of a teacher. I think this format allows educators to quickly identify and address concepts students struggle with.