We got twin Kelloggs for you today.
First is Greg, who came up on the blog the other day. Greg’s a biologist, and he posted a video debunking some bogus covid science claims that are floating around on the internet. Actually, Greg has a whole Youtube channel with bioinformatics explainers. So that’s cool. Dude can sing too.
And this got me wondering what David’s been up to. A quick google turned up this article with Cary Moskovitz, “Inquiry-Based Writing in the Laboratory Course.” Hey! I don’t teach laboratory courses, exactly, but I think statistical methods is close enough. Moskovitz and Kellogg share some insights:
The inadequacies of the traditional lab, in which students go through the motions of laboratory work in a series of “cookbook” activities, have been widely recognized. . . . However, educational reform has yet to overcome the inertia of the traditional school “lab report.” Even in inquiry-based settings, such lab reports remain largely inauthentic and make-work affairs, involving little actual communication beyond the implicit argu- ment for a good grade. Real scientific writing, on the other hand, involves a variety of rhetorical functions including persuading skeptical audiences, constructing interpretive frameworks, refuting the work of others, and so forth.
To that list, I’d add “refuting one’s own work” and “exploring the conditions for one’s work to be refuted,” but I agree the general point.
They also emphasize the importance of defining the audience for a writing task. I agree with that too.
From the other direction we can consider the mess that is scientific writing in the real world. I think Moskovitz and Kellogg are talking about undergraduate students, but things are just as bad, although in different ways, among Ph.D. students and postdoctoral trainees, where there can be huge pressure to publish in so-called top journals, a pressure that can overwhelm all other goals of communication and learning. We need to work on this at all levels, from the empty “five-paragraph essay” on King Lear in high school to empty journal articles produced by working academics, and everything in between.
So thanks, Kelloggs, for giving us lots to think about today. And happy Thanksgiving!
Kelloggs: Not just a cereal, food for thought.
I don’t share this skepticism about “traditional” lab reports. Obviously, when you do the cooling experiment in chemistry that shows how heat decays over time, you’re not going to discover cold fusion. So what? We don’t need a bunch of pretending that people are discovering things in their first-year chem labs, much less lab reports where students show their rhetorical wit and persuasion skills! My goodness! What students should be doing in their first two years of undergrad science curriculum is busting their butts to get on top of the massive pile of knowledge and skills they lack, not crying because they’re not making monumental discoveries and their lab projects aren’t interesting enough. Boo freakin’ hoo. If at the end of first year chem/phys/stats a student can write a basic lab report clearly explaining the methodology, results, and implications of the results, that’s a HUGE win.
The traditional lab report *is* a “make-work” affair, and it is so for good reasons: students need to **practice** to drill their messy little brains into patterns of scientific thought, rather than dreaming up idiotic new quackery about power poses and defending their irrational claims with more irrational rhetoric. If we could get back to discipline in education, maybe we’d have a lot fewer people sucked into blatantly implausible things like continuously filling soup bowls.
I strongly disagree with this, and I think you’re mis-understanding the criticism of cookbook labs. The ideal is *not* that students do novel research in an intro lab course. It’s that students think about how to use tools to demonstrate, for example, Newton’s law of cooling, see how the data are scattered about an expected theoretical form, recognize experimental problems and troubleshoot them, etc. Imagine, for example, being given a hot plate, thermometer, etc., and just being told: measure the cooling of something, and describe your results in the context of Newton’s law of cooling. In the a cookbook approach one mindlessly goes through a series of tasks, not really learning about the topic (beyond what one would get from a lecture or a simulation) or how to do science. The instructions are: press button A, B, and C, record 10 values of X and Y, and only X and Y, make a graph, hand it in.
Raghu says:
“In the a cookbook approach one mindlessly goes through a series of tasks, not really learning about the topic”
I guess we have a different idea of what constitutes the “traditional” lab. There’s nothing “cookbook” about the “traditional” lab in my experience except the way students are taught to do the work. I never had a lab where the instructions “button pushing” instructions, nor did I ever teach or TA such a lab.
“see how the data are scattered about an expected theoretical form”
Every lab I ever did – physics, chem, bio and geo – was like that. Students get plenty of practice recognizing the imperfection of real-world data if they do anything at all. In my experience teaching that’s the first thing students notice about any lab: that it doesn’t work perfectly or sometimes not even close.
> I guess we have a different idea of what constitutes the “traditional” lab
It sounds like your experience was the kind of experience that Moskovitz and Kellogg want more people to have. I’m wondering if this was because your classes were structured differently than what others view as “traditional” or if, like Daniel, you brought enough of your own enthusiasm to the table to make even humdrum coursework insightful/valuable. Would you elaborate on what design elements of the courses you think were most effective?
To be clear, I’m not asking this as a “make work” exercise for you, jim! I’m asking because designing these kinds of exercises is part of my job and I will gladly steal any good idea I can get my hands on.
“To be clear, I’m not asking this as a “make work” exercise for you, jim! ”
:)
I didn’t have any unique course designs. My undergrad was 100% traditional. I went to a smaller tech/engineering school though. No humanities majors. I think alot of the story was just the students’ expectation that the work would be difficult and rigorous and the profs / TAs expectation that people would do the work. Profs / TAs invested substantial effort in course prep.
In some respects my small-school undergrad experience was unusual. But OTOH when I TA’d in larger schools it didn’t seem that different. Really in my mind it’s just about establishing high expectations of the students, then giving them the tools to meet those expectations (with adequate material and clear instructions etc). I think as an instructor you have to make it visible that you’re putting as much effort into their learning as they are. If you do that you get buy in. If you don’t, people will blow you off and the quality of your materials is irrelevant. You have to get that buy-in.
Thanks for the info! I strongly agree about the importance of both student expectations and instructor investment. I feel like much effort goes into jolting students out of their maladaptive expectations. As my high school math teacher said, you don’t go to college to “get” an education, you go there to *take* an education.
“you go there to *take* an education.”
:)
Raghu:
I guess summarizing, my view is that most basic science teaching materials have been sufficient for a long time. I mean of course there are some terrible books lab books and other stuff but other materials that have been used successfully for decades sometimes repeating the same format in different books with only minor modifications.
My view is that a student is **by far** the most powerful agent in their own learning. Once the sound basic materials are in place it’s up to the instructor to tap the students’ drive – to get them to buy in to the idea that what you want to teach them is important enough for them to invest the effort in learning it.
My guess is that you’re successful teaching because you convince the students it’s worth the effort, not because of the materials or specific approach you use. You care about education and that’s a big factor in convincing students to work. Probably the same for many people who write here.
The upside of this approach is that it doesn’t require a bunch of work developing new curriculum. It requires one skill – pitching the curriculum – that can be used across all of academics and in almost everything else people do. It’s a very generalized skill. And in the process of doing it you’re getting the added benefit of showing your students how to do it.
It’s incredible that you can say something that makes so little sense without noticing it. It’s exactly backwards.
Our problem isn’t that researchers are insufficiently adherent to conventions. People are very good at convincingly writing up formal lab reports on methodology they’ve copied from somewhere else and only partially understand. Power-pose nonsense was written with all the superficial “discipline” of an excellent school lab report. It had the headings and the LaTeX and the figure labels and the Methodology and Discussion sections and the data tables–judging things by the standards you advocate here, it’s good science! People are pulled into blatantly implausible things because they figure if the write up is very clear and disciplined by the standards of the scientific authority, and if they follow the methodologies handed out by the gods, then it’s a good paper.
As Raghu pointed out, it’s not about making new discoveries. It’s about being able to synthesize a workable methodology. Can you design an experiment to measure the gravitational constant without being told how? If it doesn’t work, can you figure out why? At least in physics, right now if get a number you don’t expect you in a lab you just more carefully reread the instructions.
As it is, understanding how experimental methods connect to theory and thinking critically about their weaknesses is typically something you pick up in your research assistant jobs. On the job learning is great, but it’s also not necessarily the most efficient. For one, this means there are no standards–it’s mostly up to whoever happens to be your PI. For another, it can be hard to learn when working on questions where nobody really knows the right answer. If you try to measure the gravitational constant, if you get the wrong number you know the fault lies somewhere in your setup. Most research labs don’t work on such settled questions.
“People are very good at convincingly writing up formal lab reports on methodology they’ve copied from somewhere else and only partially understand. ”
Exactly! they’re copying. You can’t write a lab and design exam questions from the labs that show that students know – and can clearly express their knowledge of – the material? That’s the job as a TA or teacher and it always has been. I don’t think that’s new or “untraditional”.
“understanding how experimental methods connect to theory and thinking critically about their weaknesses is typically something you pick up in your research assistant jobs. ”
That wasn’t my experience. As far as I knew that was part of the traditional lab. You’re supposed to evaluate the results – which includes assessing the validity of the assumptions and discussing possible experimental failures. That’s what we were taught to do. But maybe if you’re not teaching students about the assumptions involved in an experiment?
“Can you design an experiment to measure the gravitational constant without being told how?”
I interpret this to mean people should be able to design experiments in their discipline. I think that’s a tall expectation for 1-2yr undergrads – for the most part they just don’t have enough knowledge design a sensible experiment except in the simplest circumstances. That’s why they have to get a PhD to become a researcher! :)
I don’t know when you went through undergrad, but in my experience a few years ago, undergrad labs meant you got a thick packet of extremely detailed diagrams and instructions, often with photos of the previous setup, and a description of what you’re meant to see and why. The primary difficulty was an exceedingly large quantity of labs to complete in a semester, and grading was harsh with respect to superficial (but important!) details like captions on figures, having the right sections, units in data tables and inclusion of all relevant equations. Otherwise, doing well was entirely straightforward. If you knew how to write in the house style, could paraphrase the instructional packet, and had a command of some kind of typesetting tool, a good grade was just a matter of time. And of course, you had the perform the bullshit ritual of “propagation of errors”, where you tacitly assume independent normal errors and apply the delta method without learning any of those words or where they break. This was true across physics, biology, and chemistry at multiple well regarded institutions.
“Evaluating results” is trivial in this setting. The results are always exactly what you expect, because you’re required to copy exactly the same design. “Experimental failures” are obvious–it’s always something like “friction, air resistance, EM interference, 60 hz radio waves from the building’s wiring”. We could run it on teflon, do it in a vacuum, build a faraday cage. Just textbook recitation–talk is cheap. Actually trying to do it and, crucially, seeing what new errors you introduce in the process, is the rub.
Right, and once they have their PhD, this knowledge is spontaneously implanted into their brain! Realistically, they pick it up on the job while they’re graduate research assistants, or maybe they don’t and they just remain bad at it and nobody notices.
The simplest circumstances is what I’m asking for. I’m talking dropping stuff and timing it to measure acceleration. Oh, there are big errors, maybe we can try dropping from a greater height? Now try using a perfect sphere with a smooth surface and a known drag coefficient? Or maybe we can try a ramping up the current through a solenoid until a magnet of known mass starts to levitate? Designing a simple experiment and improving on it iteratively is harder and more educational than copying the EPR experiment.
Your experience in labs was much different than mine. For phys and chem we had department-provided lab books but nothing like you describe. My experiments were rarely perfect. We had a TA introduce every lab, supervise the lab work and grade the lab reports. For chem there was a full-time lab overseer as I recall that supervised TAs in grading. For phys it was a crap shoot, many of the TAs barely spoke english. Bio lab reports were the most rigorous. I went to a small engineering school but things looked pretty much the same at the larger schools where I did grad work, but I wasn’t involved in other disciplines.
I agree with you, if the lab is so cook-book that you can’t go wrong, then it’s a bad lab. But as far as I know that’s not the traditional approach. That’s the new directive-from-admin “nobody fails” approach that keeps butts in the seats and keeps those state dollars flowing.
The way you describe it is pretty much how it was for me. We just disagree on the value of it. Indeed, we had a TA introduce, supervise, and grade, and things went wrong all the time–we worked with lots of very complex, expensive equipment. The problem is that that there’s a simple algorithm to fix it. If you follow the packet from beginning to end and check every step and spend enough time, eventually it’ll work. The problem is that following a very complex and difficult design from somebody else is categorically not the same as designing your own experiment.
And your hypothesis makes no sense to me. For one, if you’re looking for grade inflation, you really want to look at private schools rather than state schools–that much is strongly corroborated by both quantitative evidence and the subjective experiences of those who have attended both. And in my case, the same exact person had been teaching the exact same course from the 70s up to a semester or two ahead of me, from mostly the exact same handouts. I can say for certain the curricula were decades old because when they came up with a lab design, they would publish a paper on it as an educational experience which was essentially our handout paraphrased. For example:
https://aapt.scitation.org/doi/10.1119/1.12419
So yes, I most certainly did get the “traditional experience” and I got the highest marks possible and I learned very little of value.
“If you follow the packet from beginning to end and check every step and spend enough time, eventually it’ll work. The problem is that following a very complex and difficult design from somebody else is categorically not the same as designing your own experiment.”
Well first of all I have no idea what level of lab your referring to. First year undergrads don’t need to be figuring out what experiments to run! Yes, absolutely, you should be able to figure out how to do the experiment from the lab book, that’s the whole point of the lab book!! What the heck is it that you expect? The job is to set up the equipment, measure the input materials properly, run the experiment and collect the data, analyze the data, generate the figs and tables, write up the methods, analyze the results and come to some conclusions. That’s plenty for first year students.
“We just disagree on the value of it.”
No, you disagree on the value of it to you personally. While in grad school, I TA’d probably 30+ labs at two different institutions, at least six different courses up to 400 level and also taught community college intro classes. I mean hey, if your parents are nuclear physicists at Los Alamos, you probably don’t need this work. If you’re a genius, you probably don’t need it. But most students need this work. They don’t know how to do the basics. They need practice doing it.
But they also need instructors who actually *make them do the work* – properly analyzing and writing up the methods and using the data that the generated themselves to write the report. If instructors keep telling them “oh this is stupid don’t waste your time with it” they’ll believe and stop bothering, just crank out the minimum trash and be happy to walk with whatever you give them.
Going to just echo Ragu and “somebody”. I would dearly DEARLY love to make a living teaching undergrads how to really actually do scientific thinking and measurement in the context of “non discovery” (ie. validating known stuff in the lab), including design of experiments, data analysis, and statistics.
As far as I can tell there are 3 barriers to this:
1) Departments don’t actually want to do this
2) Students want cookie cutter things that will get them guaranteed good grades if they follow the steps
3) Even if someone wants it, they’ll pay a fixed dollar amount that amounts to less than minimum wage to adjunct immigrants who need to stay in line to keep their H1B visa status.
I LOVED my undergrad engineering labs. I got terrible grades on the lab reports because I treated them as actual challenges to do real experiments. I wrote up my reports in LaTeX with equations you could read, I tried out alternative statistical techniques, I did least squares regression instead of plugging into formulas… (I was 27 or so, I had a lot more knowledge than the 19 year olds).
Later the professors pulled me into their offices and said “can you just do the writeups the way we asked because we dont’ want to give you a bad grade, and also maybe if you’re going to be here next year can you be the head TA for our labs?”
There is so much potential joy in actually learning how to do science. Sad that we crush it like a bug.
“2) Students want cookie cutter things that will get them guaranteed good grades if they follow the steps”
Yes, they do. Administrators want that too, so that they can follow the cookbook when the student comes and complains that they weren’t graded “fairly” because they didn’t have an instruction sheet on how to get an “A”.
Its the job of teaching staff to push back against this.
In my experience no one cares about teaching at Universities. The administration for reasons you mention, the professors because it counts against them as time not spent researching, the adjuncts because anything they do to rock the boat puts them in danger of losing the car they’re living out of, the students because they’re only there to signal to potential employers, and the parents because they only care about whether their students can pay off their loans after school.
It’s a disaster of epic proportions.
Superb summary. +10
I didn’t teach any lab courses, but gave homework assignments, and gave out a handout on what I expected them to hand in. Here is an example from a graduate course in regression analysis (Item #4 is the most important; the others are secondary. ).
“I will expect you to write up your homework solutions carefully. Do not hand in a rough draft. In particular:
1. Write in complete sentences.
2. Organize your presentation. In particular, put computer output and graphs as close as possible to the place where you discuss or refer to them. (Please do not put them at the end of each problem.) This often requires cutting and pasting (either by hand or computer). In some cases, writing on your computer output will work.
3. Do not hand in computer output that you have not referred to in your discussion. Again, this may require cutting and pasting. But be sure to include computer output that you have referred to in your discussion.
4. Explain your reasoning clearly. The quality of your reasoning will be an important consideration in your grade, especially as the semester progresses and you have more options available to consider. Do not expect full credit if you do not give reasons for your answers or if you do not interpret your output in the context of the problem.
5. Write legibly. ”
Also, even though I had a TA to grade homework, I always graded about three papers myself, then made up a “grading rubric” for the TA to use in grading the rest of the papers, AND met with the grader to go over the papers I had graded, to show her how I expected her to grade the remaining papers.
That’s quite clear! I would have liked your course!
“There is so much potential joy in actually learning how to do science. Sad that we crush it like a bug.”
I’ve heard this so often for so long I can’t help but wonder if this is a “zombie idea” that substitutes for the reality that most students just don’t want to work very hard to learn science.
“I got terrible grades on the lab reports because I treated them as actual challenges to do real experiments…”
Obviously, since you were invited to be head TA, you were doing great work. Just not the assigned work. The assigned work is there for the bulk of the students, most of whom need an assignment below your level of skill and enthusiasm. I’m a big believer in teaching to the top 20-30% rather than the median. But if you teach to the top 1% you’re just not teaching enough people. Those people have to provide their own enthusiasm.
I don’t disagree with you about what level to teach to, but the assignments were aimed at the 25%tile or less. You just had to copy down what the TA wrote on the board, and fill in your own numbers, make a graph in Excel and don’t forget to label it. There was no real thinking involved.
Of course i could bring more enthusiasm and etc but the fact was the existing stuff was so mindless that even people who didn’t understand a thing about what they were doing got an A. This was 3rd year undergrad Engineering majors.
Let me give you an example. Mix some clay with water in a tall graduated cylinder. Float a hydrometer in the water, and take readings of the fluid density through time for 2 hours. Plot it and turn it in. Effectively this would get you an A even if you didn’t understand the whole explanation (and the TA did in fact give some kind of explanation of why we were doing it, at least that!).
Now in my opinion this is trivial and any 5th grader could do it. That was pretty close to what our lab required.
Instead suppose you give them a scale drawing of the hydrometer in the graduated cylinder, and have them draw a free body diagram of the device immediately after mixing the clay… then again at 1 hour, and again at 2 hours. Mention that the clay particles settle through the water due to gravity and that their specific gravity is usually about 2.7. Mention that settling rate rapidly asymptotes to a constant that is a function of drag which is a function of diameter of the particle and give a dataset of settling rate vs size and a plot of that data.
Now, perform the experiment of mixing the clay with the water and take the data… Using the tools of your choice, write a report in which you describe what you are able to learn from the data about the distribution of particle sizes in the clay (give the students the opportunity to think it through).
After writing the report and handing it in, have the students write a computer program that discretizes the particle distribution into 20 sized particles and 20 starting depths. Using Matlab/Julia/R, and an assumption of constant sink-rate per particle size, solve for the mass of particles at each height after 1 hour and 2 hours. Using this simulation, calculate the density of the fluid at each height. Calculate the fluid hydrostatic pressure at each height. By summing the hydrostatic pressure calculate the buoyancy force, and compare it to the total buoyancy force that would be exerted by a fluid of a fixed density. Solve for the equivalent density reading on the hydrometer after 1hr and 2hrs. Compare to the readings taken.
Those things were within the capacity of the students i studied with. But that sort of working out the basic model of what was going on wasn’t emphasized in the labs.
I can’t speak to what goes on in engineering labs. I’m not an engineer.
just the same, this:
“Instead suppose you give them a scale … and a plot of that data.
Now, perform the experiment of mixing the clay with the water and take the data… Using the tools of your choice, write a report in which you describe what you are able to learn from the data about the distribution of particle sizes in the clay (give the students the opportunity to think it through).”
is how most of the labs I worked on were written. To me this is a traditional form of lab assignment and was common in published lab books and text books when I was in school and teaching, which wasn’t *that* long ago! :)
I had one class in grad school where I failed my first assignment because I had entered one cell wrong in a spreadsheet. Geostatistics! :)
I have had good success in physics labs combining a few ideas that others have given me. 1. Expect more of the students. Maybe the student is given the hypothesis, and maybe not, but at least the student should design and run the experiment, acquire the data, analyze the data, come to a conclusion, and write up the lab report from scratch with good grammar and usage. Totally agree that the cookie-cutter approach is severely lacking. I got all the way through a Ph.D. in Mathematical Physics without a single physics lab like this. The only really good labs I had were Electrical Engineering! 2. Most students will not understand why they need to do all those steps, so model it for them. I did that on the first lab day: went all the way from hypothesis to writing up the lab report right in front of them. I explained what I was doing and why the whole time. I had one student take notes while all the other students just watched. The quality of the lab reports I got jumped dramatically when I did that. They understood why certain sections needed to be included in the report, and also sometimes when they could be left out. 3. True statistics, the backbone of modern science, is extraordinarily difficult to teach because of a lack of time required to repeat experiments enough times. Workaround: have some extremely quick “running experiment” throughout the semester (for E&M Physics, I had students measure resistances to see if they were within tolerance) that they can set up, get 5-10 measurements, then take down, all in 10 min or less. Then have them do a full statistical analysis for that experiment at the end of the semester.
This process enabled me to truly teach experimental science in a way I never got until I finished my degree and started doing experimental science.
My understanding is that currently, “lab” classes at the undergraduate level have for some reason become a kind of “advanced topics in circuits, with demonstrations.” Plus a bit of terrible LabView programming. I think the problem is partially that there’s so many topics auxiliary to physics that are nonetheless necessary for experimental research that it crowds out the real issues of experimental design and measurement. There are only so many units in an undergraduate courseload. At the same time, I can’t shake the feeling that part of the problem is that a class on experimental design and measurement would necessitate more time and interaction with students than professors are willing to put up with.
> it crowds out the real issues of experimental design and measurement
Oh actually this makes me wonder is it obvious that lab classes should be the one with the writing component?
It’s an accomplishment on its own to get the experiment running/thing built a lot of the time lol.
This is an interesting discussion, to which I would like to add a pitch for pluralism. My experience is that people learn and think in different ways, and it is hard so say that one is better than another. As an example, many decades ago, when I was between college and graduate school, I tagged along with a friend to a few graduate seminars in neurophysiology at Standford. The professors were Don Kennedy (later president of Stanford) and Don Wilson (who died prematurely in a boating accident). They had very different styles. Kennedy was more methodical, thinking about experiments that would make incremental advances. Wilson was more speculative, trying to identify critical experiments that would move the field in one direction or another. Both were good scientists, and I think science needs both kind.
Well, this is just delightful, and I’m sorry I missed this last year. For what it’s worth, David and I did a collaborative project actually trying to implement the model he and Cary proposed. The lab reports are very challenging to mark, but the learning experience of students is just fantastic. I use the framework in some advanced undergraduate and graduate classes as well. https://pubmed.ncbi.nlm.nih.gov/32746505/