They learn how to fit multilevel models

Robin Hanson is skeptical of my response in the following exchange:

Hanson: What do the customers who are paying your salary get from you?

Gelman: They learn how to fit multilevel models.

He accepts that my response (which he considers “an idealistic theory”!) to be “pretty plausible from an evolutionary point of view” but says it has “detailed problems.” In his words:

College students prefer to be taught by profs who research, and hence ignore students more, yet students have little idea what their profs research. . . . There is relatively little relation between what profs teach, what profs research, and what students do after they graduate.

To which I reply: No way, dude! Our students do know that I research multilevel models. They send me emails asking when I’m going to teach multilevel models and Bayesian statistics. And, if they can’t take my class, they ask what other classes I recommend. And, yes, there is a strong connection between what I teach and what I research. And it’s my impression that they do use this stuff after they graduate (although I agree I don’t have survey data on this, and that this lack of followup is indeed a weakness of academic organizations).

You might say: Fine, but Robin is talking about academia in general, not the Columbia statistics department in general. That I could buy–in fact, I posted a note last month to that effect: econ undergrads at George Mason University may be very different from statistics and political science grad students at Columbia University. But . . . in his blog entry Robin appears to be skeptical of my claim that the customers who pay my salary “learn how to fit multilevel models.” He seems to think this is “a story” and “an idealistic theory.” I think it’s close to the truth. Just because Robin is being cynical, it doesn’t mean he’s accurate.

Robin disagrees with, in his words, “the standard idealistic view . . . that academics know useful and important things, things which students want to learn, media want to report, consulting clients want to apply, and which foundations and governments want to promote the creation and spread of, for the good of the everyone.” With which parts of this statement does Robin disagree?

1. “Academics know useful and important things.” Certainly, some of us do. Others are useless time-servers. Similarly, some dentists are great, whereas others will wreck your teeth. It’s a good idea to choose carefully, perhaps look over an academic researcher’s published writings, to make your judgment.

2. “Things which students want to learn.” Yes, things such as multilevel models.

3. “Media want to report.” Unfortunately, lots of sensationalism here. I don’t really feel like defending the news media (or even blogs) right now.

4. “Consulting clients want to apply.” See item 2 above.

5. “Which foundations and governments want to promote the creation and spread of, for the good of the everyone.” Pretty much, yeah. I’m not here to defend the goals of (say) the state government of New York or the Heritage Foundation or whatever, but I assume they would like their conclusions to be research-based, to avoid negative unintended consequences and all the other things that we worry about when considering policies.

Calling “the standard idealistic view” is a nice rhetorical trick, but I don’t think it shoots down my claim that students are taking statistics classes for a good reason.

One more thing

I was amused to see that Robin misquoted my “I’m glad I’m not an economist,” changing it to “I’m glad I’m not a social scientist.” He then elaborates that I “apparently thinks that having to attend to the details of social phenomena consistently forces social scientists to accept distasteful views on many social topics.”

Huh? I never said this. Robin believes that people have stated reasons for doing things and “actual” reasons, and the actual reasons, in his opinion, often turn out to be things that I consider silly, such as the desire to be credentialed. I don’t think that having to attend to the details of social pheonomena “forces” Robin to have his views. I think these are views that Robin has, that he shares with many economists, but that are not so commonly held by researchers in other fields.

Still more!

Above I noted that Robin described m statement,”They learn how to fit multilevel models,” as “pretty plausible from an evolutionary point of view.” But now I noticed he changed this to “pretty implausible . . . ” Huh???

I guess I don’t really know what the “evolutionary point of view” here is. But it all seems plausible to me! I offered a course on multilevel models, it was popular, I wrote a book, more people want to take the class, people are bugging me to find out when it’s offered next, etc. What’s “implausible” about that?

15 thoughts on “They learn how to fit multilevel models

  1. I think I take issue with "College students prefer to be taught by profs who research, and hence ignore students more…" more than anything else. Yes, it's important that profs are still active in their field, but what's meant by "ignore students more"? This current big U culture where TAs teach all courses and students never see or interact with their profs is rather ridiculous. There's a need for much better balance than that.

  2. Seth:

    That's in fact what I wrote in a comment to my earlier blog entry which Robin linked to. I wrote: "One issue, I suppose, is that the credentialing for grad students is much different than the credentialing for undergrads; maybe I was thinking more of the former while you were thinking of the latter. (We have very few undergrad statistics majors at Columbia, and I don't interact much with the undergraduate poli sci majors. But I spend lots of time with grad students and postdocs.)"

    But in his blog response, Robin avoided taking this split-the-difference option and specifically disagreed with my "They learn to fit multilevel models" response. So I think Robin thinks his description holds for grad school too.

  3. I was talking about college students (i.e., undergrads) when I said teaching has little relation to research. And I made a typo re "implausible." Yes you may be assigned to teach a grad research topics course on one of your research areas. But it seems you just assume the idealistic story, that your stat tricks are just directly useful to your students and patrons, without having checked much.

  4. I am not sure what is behind all of this but I sometimes get the comment from colleagues and students that profs should concentrate on teaching and abandon research since the teaching is what the students pay for.

    My reply is to ask when they would have liked this policy to be first instituted? How about 1451 which is the year the Glasgow University charter was granted? Surely the students then had as much right as those now to have their profs devote all their energy to teaching.

    Suppose now that Glasgow University had implemented this ideal policy in 1451, what should we now teach? The science of the 15th century? Well no, I guess we would choose to teach the science that had been discovered in those other universities who had not implemented this policy. They no doubt would be the universities, rather than Glasgow that would have allowed Joseph Lister to develop antiseptic surgery, William Thomson to develop thermodynamics and Adam Smith to develop his economic theories

    I can't help thinking that given a choice between studying in a university in which nothing was researched but the academics merely learned what others had first understood to pass it on down the line and one in which there was a full participation in the ongoing adventure of education, students would opt for the latter. Thus, I am glad that it is the policy of the the institution in which I work that both research and teaching are to be promoted and I can't see the point of arguing that they should be separated.

  5. Stephen: I agree with you completely, but, interestingly, Robin's comment is the opposite. He says that everybody says that students are paying for teaching, but that actually students are paying for proximity to bigshots (even, presumably, at less prestigious universities that, even if they don't have a lot of bigshots, at least still have some profs who graduated from Harvard, Yale, Michigan, etc). He also has a side point that our research maybe isn't so great, but, to be fair, he's not making the conventional argument that we should be spending more time teaching.

    That said, I find that a bit of teaching (we do about 4 courses a year) is good for my research.

  6. Robin:

    1. I think we can agree that undergrads and graduate students have different levels of focus on their curricula. I think undergrads come to learn also, but they don't in general know as much about the course topics, and more often they will choose a class based on a general catalog description rather than specifically because they believe the class will be useful to them.

    2. My multilevel modeling course is a graduate class with students from several different departments, but it is not "a grad research topics course" as such a thing is usually defined.

    3. I still don't see what's "implausible" about my story: I offered a course on multilevel models, it was popular, I wrote a book, more people want to take the class, people are bugging me to find out when it's offered next, etc.

    4. You write "you just assume the idealistic story, that your stat tricks are just directly useful to your students and patrons, without having checked much." I guess it would help if I did a survey (rather than, for example, posting blog entries), so I don't know. But why should you be so quick to assume that the thousands of people who've bought our books and the hundreds who've clamored to take my class do not find these statistical methods (I'd prefer not to use the somewhat belittling term "tricks") directly useful? Why are you so sure that, just because my course is taught at a university, that it's irrelevant? I recommend that you consider the hypothesis that some views that you disparage as "idealistic stories" might actually be true! You could also ask your colleague Alex Tabarrok his view on whether statistical modeling can be useful to people after the graduate.

  7. I took Andrew's course last fall to learn multilevel models. I graduated in May and have applied them several times in real world policy research (although the "real world" does seem a bit uncomfortable with multilevel models unless they see a similar OLS model with dummies and interactions too).

    This is purely anecdotal but I don't think Andrew is hopelessly idealistic. In some areas Robin's got a point too, though…I'm not sure how much specific use I'll get out of deriving the Romer growth model or proving that OLS is BLUE. At best, just a generally feeling that it makes you smarter.

  8. Robin needs to wake up and smell the predictive modeling. Whether or not you get your next car/home loan is completely dependent on a multilevel model. At just about every big-box, retail store, I am begged to be included in their "rewards" program (or, "give us your preference data so we can model the heck out of it in order to figure out how to extract more of your money" program). My neighbor just got a free home security system with no monitoring fees in exchange for him sending in all of his gas/grocery/consumer products receipts on a monthly basis. At a previous job, the results of a predictive model I worked on influenced the decision to put certain patients in a more intensive care program.
    So perhaps Robin just doesn't realize how important modeling has become in corporate and institutional decision-making. It's gotten to the point where I seriously think that modeling should be taught in high school AP classes.

  9. @Robin: Multi-level models and Bayesian approaches to inference are not tricks nor explicitly Andrew's. They are useful tools for answering many research questions, a large subset of which have direct policy implications. Some scientists use these tools to communicate informative results to policy makers (perhaps because they work for a public/private sector agency that paid them to do so) and some do not.

    But I don't think that is your point. If I understand you correctly, you are making a critique about the return on investment in higher education. Ie, undergrads pay lots of money and take 4 years out of their lives to sit in a classroom and let Andrew teach them abstract statistical tricks that will have no bearing on their future careers? On top of that our tax dollars (NSF, NIH money) goes to people like Andrew who only publish esoteric stats papers in obscure micro-journals with the sole purpose of furthering their own careers?

    Your point is valid but cliched and misdirected. Cliched because, well duh, that's academia…
    Misdirected because Andrew is part of the solution, not the problem. I've never taken his classes but his book with J.Hill is one of the few (if only) accessible, pragmatic stat guides out there ('Mostly Harmless' is fantastic but doesn't have code and targets a narrower audience) .

    They use narrative examples and code rather than terse statements and lengthy proofs. The book is cheap, paper-back, and has open-source code snippets. They also have questions/resources, et cetera if you want to teach from it. Prior to academia (I'm a grad student) I spent several years doing analytical consulting work, never touching multi-level models, but I still would have benefited greatly from that book. I have friends in industry that use it and have met others who don't use MH models or Bayesian inference but still use the Gelman-Hill book as an applied reference and teaching tool.

    Thus, though my evidence is anecdotal (@Andrew the survey is a good idea)I think that Gelman and Hill have many satisfied clients. I'm curious to hear what others think.

  10. This seems fairly simple to test in a rough way. Don't universities keep course evaluation survey data from students? For non-required undergraduate courses, or required courses with multiple sections students can choose between, compare grades and "this was an easy course" versus "I learned a lot from this course" to see which better predicts enrollment and professor/course ratings. If undergraduates are more influenced by easy As than learning, that suggests credentialism and association rather than an interest in education.

  11. For two reasons it is a big mistake to focus on multi-level models and Prof. Gelman's teaching. 1. Prof. Gelman is among the most practically oriented professors in an arguably more practically oriented field.
    2. It is harder to be objective about something so close to home.

    I think it is clear that people choose colleges and courses of study for many reasons including:

    1. learning useful things
    2. learning fun things
    3. signaling that they are smart and hard working
    4. to associate with students and faculty who are smart/hard working/wealthy/impressive

    Similarly, we perform, and fund, research for various reasons.

  12. Also, Zubon makes a good suggestion. Related research has been performed. I believe it shows undergrads seek out classes which provide better grades. Course evaluations are also predicted by professor attractiveness, and even better by 30 second video clips.

  13. Speaking as a former "college student" and former grad student, it seems that no matter the motivation, results are results. Supposing that all I wanted to do was get credentialed, the fact is that now these tools are useful to me. For example, I can follow you argument, I know what a multi-level model is, and I can decide for myself who is wrong. Useful, no matter the motivation.

Comments are closed.