Above is the title, and here’s the abstract:
The logic of social science can work in two directions: generative modeling predicts behavior given assumed preferences, and inferential reasoning deduces preferences given observed behavior. Both these modes of reasoning can be valuable, but in choosing which mode to use, social scientists have the freedom to come to essentially opposite conclusions for any problem that comes in. We give examples from economics, political science and public health. On the plus side, recognition of this danger has the potential to improve social science for those researchers who come to terms with this disjunction and move to a more holistic integration of data and theory.
Now I just have to write the paper. The idea is to have a cleaned-up version of this post from a few years ago and various related material. And here’s a good example from Tyler Cowen. Just to be clear: by calling that “a good example,” I’m not speaking ironically and saying that Cowen’s post is a good example of mistaken thinking; rather, I’m saying it’s an example of a good discussion of the problem, and it’s an example I’d like to use in this as-yet-unwritten paper.
Well, they shouldn’t give opposite conclusions, should they? For example, to study the effects of mask wearing, the deductive approach will make some assumptions about how wearing masks impacts behavior (as well as assuming some level of adherence and effectiveness). The inductive approach should examine data on how actual mask wearing has affected both disease transmission and other behavior (including diminished enjoyment of activities). It would not be surprising for the two approaches to lead to differing, perhaps even opposite conclusions regarding the welfare effects of mask mandates.
But, if the conclusions differ, doesn’t that mean the deductive approach made assumptions at odds with actual behavior? This is fairly common for economic studies – we often assume people should behave in ways that differ from how they actually behave. It is instructive to find this out – but I don’t think it really says that either approach is superior to the other. The advantage of the deductive approach is that it provides additional structure that the inductive approach may lack. But if I have to choose between the two, I’ll go with the inductive if it differs markedly from what a deductive analysis would suggest. My primary learning would be that my deductive assumptions were incorrect.
I never really liked the concept of inductive reasoning. Pierce’s concept of abductive reasoning (i.e. guessing) seems to better explain what people do:
https://en.wikipedia.org/wiki/Abductive_reasoning
And you need to use both abduction and deduction, they are not in competition at all.
First, you observe something. Second, you abduce (guess) one or more possible explanations for the observation. Third, you deduce what else should be observed if your guess is correct.* Fourth, you go make those new observations and compare to the deduction.
It seems straightforward to me but there seems to be a lot of resistance to following through this entire process. People seem to prefer doing only one or two of the steps and skipping the others.
* ideally, these predictions should be “surprising”, i.e. unlikely under the other proposed explanations. This is where the denominator of bayes rule comes from.
This is not really in response to you – it just spurred something I was looking at today. One of our favorite researchers, Brian Wasnink, is pursuing an active web-based life after his teaching career was cut short. Among his endeavors is advice for academics, some based on his experience with retracted work. Among his pearls is this one: “Two of these steps include 1) Choose a publishable topic, and 2) have a rough mental roadmap of what the finished paper might look. That is, what’s the positioning, the study, and the possible contribution.” Viewed in one light, there is nothing objectionable here. But (and this is the tenuous link to your comment), potentially the “First, you observe something” comes close to “First, know what your paper is going to say…” I’d venture that as one of the biggest dangers of deductive reasoning.
Dude’s got chutzpah, I’ll grant him that. But I won’t go so far as say, “You gotta hand it to him,” since over the past decades he’s wasted millions of government and corporate dollars and wasted untold hours of people’s time. Not to mention that he made the Cornell public affairs office look like a bunch of fools. Not cool.
https://twitter.com/dril/status/831805955402776576?s=20
lol i knew this was coming
The original motto of science was nullius in verba. Someone could waste their time/money producing all the pre-determined “observations” they want, it would never grow to a problematic level if independent replication was standard.
But the reliability of the observations is obviously crucial. It is a waste of time to abduce explanations for a bunch of incorrect facts.
Couldn’t the economist just reply “yes, people are often rational, optimizing in surprising ways; but sometimes they fail to optimize and that’s where we can help.”
Formally, yes, one could always go in either direction for any given problem. But presumably the specific details help you decide between “people are optimizing here” and “people aren’t optimizing.”
Dmitri:
The problem is that it’s like the faces-vase thing. I’ll see an economist making a generative argument (of the form, “People are acting suboptimally. They should follow our advice and invest in index funds or go for it on 4th down or charge more for restaurant meals or fire a bunch of teachers or whatever.”), in which case the inferential argument doesn’t come up at all (unless it’s a lament that these people don’t listen to economists enough, haven’t been well enough educated, etc). Or the economist will make an inferential argument (“People’s actions show their revealed preferences. Whether they’re risking their lives or playing the lottery or paying for collision damage waiver or whatever, they’re doing this for good reasons, and don’t listen to pointy-headed professors saying otherwise.”), in which case the generative argument doesn’t come up at all.
The argument either follows an elitist framing in which ordinary people are nudgeable fools who should be listening to their economist overlords who will tell them how to live rationally, or a populist framing in which economist are anti-paternalistic tribunes of the people, protecting the common man from naive social engineers. I think that going back and forth between these two frames can be helpful, both to explore how people can do better and to understand the motivations for what they are doing. But typically it seems to me that we’ll see one argument or the other but not both for the same problem. In which case it becomes hugely important which direction the economist wants to argue in any given example. See here for an elaboration of this point.
I like the idea of trying to look at both sides at once, that seems both useful and like a nice way to defuse the criticism I am imagining.
I think in the fourth down case you do see both sides. Almost every discussion hypothesizes that coaches punt because they don’t want to take the blame if things don’t work out. So they lower their chances of winning in order to avoid the appearance of responsibility.
The assumption there is always that coaches shouldn’t care about that thing. One direction of the argument, I guess, would say “well, preferences are what they are.”
In practice these approaches are often alternative or even competitive, but (in theory) they shouldn’t be. I prefer to think of them as two moments of the same general process. In a way, each contains the germ of the other: a generative analysis (deduction?) embodies assumptions about agents’ motives or other relevant variables, and an inferential analysis (induction?) embodies modeling assumptions that make it possible to generalize beyond the sample or its observed behavior.
Of course, these moments don’t have to be the work of the same people, and it’s probably better to have a division of labor, but there should be constructive communication between them. OTOH, a continuing theme of this blog is how researchers dig in when challenged. If creative correction (copyright this) is a key part of the story, it should be exactly the opposite.
Peter:
Yes, I agree.
I am clearly not hep to the jive of today: what is a “vig”?
Vigorish.
Thanks! Though I still had to look that one up.
That’s the problem with language, if you want to know what one word means, you have to look up another word! It never ends…
It is not the jive of today. :)
I think I have demonstrated that I am not now nor have ever been “hep”.
It is probably more hep to use “hep” instead of hip.
I, too, recognize this kind of paradoxical thinking in pop economics. The author picks a role–either the actor or the observer–to portray as foolish, and makes the reader feel like the author has let them into an exclusive Smart People Club. It’s lazy but effective. And yet, there’s nothing inherently wrong with examining behavior from multiple perspectives, so long as the author doesn’t impose a false dichotomy. There is a right way to do it, that presents insight without pandering or exaggeration. You might call it causation and optimization analysis.
Causation tells us what leads people to behave as they do; optimization tells us the behaviors that are likely to achieve preferred outcomes. Both approaches assume people’s behavior is a response to perceived stimuli and incentives, moderated by prior knowledge and habit. The two analyses aren’t contradictory and don’t necessarily have to be applied to cases of irrationality.
Yes, sometimes an actor’s perceptions, knowledge, and habits lead to non-obvious insights, which in turn lead to rational behavior that looks irrational to observers. And, yes, other times, those perceptions, knowledge, and habits are really misperceptions, ignorance, and stubbornness, which lead to behavior that’s recognized as genuinely irrational to a rational observer. But both actor and observer can be rational, or both can be irrational. The problem is that authors often feel they must twist the situation create situational irony for entertainment sake, sacrificing logic or facts in the process.
As a complete aside, reading the comments section of Cowen’s blog makes me really, really appreciate the community you’ve built around yours.
+1 for your complete aside. I always fear that the only thing standing between the comments here and Tyler’s is Andrew’s relative obscurity. The obscurity is unmerited, but if it keeps the comment level high, it’s a price the readers are all willing to bear, if not the writer. 15 years ago, I used to comment on Tyler’s blog… now I can’t even read the comments, much less add to the cesspool.
I agree that Marginal Revolution comments are generally awful, but there are occasional gems. This one from today, though somewhat off topic, is fascinating, for example. I sometimes try to comment there to try to offer something constructive, but I can’t help feeling vaguely icky afterwards.
J(oa) –
M not familiar with the blog – I’m curious what you find cesspoolish about the comments there?
There are a large number of posters on MR that continually are vulgar, sexist, racist, and intentionally inflammatory. In fact, they pride themselves on being politically incorrect and inflaming any sentiments they consider “woke” (whatever that term means). They also tend to be libertarian in the extreme – an extreme that is impossible to argue with and, in my opinion, adds nothing constructive to any meaningful policy debate.
Count me as one of the people that reads that blog because they occasionally post references to interesting things, but find the comment section unrewarding, and often, infuriating. I’ll admit to occasionally posting a comment – never using my real name, whereas I am happy to identify myself on this blog.
I would add that I find Tyler Cowen disturbing. He’s plenty smart, but he prefers posts that generate attention at the expense of being careful and thoughtful. For example, he publishes a book on the topic of the Great Stagnation (where we’ve run out of large technological leaps), only to claim a few months later that that era is over and marvelous things are going to happen due to BitCoin, AI, etc. It’s hard for me to take anything he writes seriously.
Hmmm, do I have this right?
a) researcher surveys preferences, models behaviour, that doesn’t match reality, concludes people are dumb
b) researcher observes behaviour, induces preferences, they don’t match reality, concludes people are dumb
My take is that neither the model nor the induction were useful, and that only research that links preferences to behaviour both ways results in understanding something about actual people.
Tyler Cowen’s take, if I read him right, is “telling people they’re dumb is a bad way to change their behaviour”, to which I add, “especially if you haven’t understood them in the first place”.
I don’t really like his Covid tax model, because reality involves a feedback loop: for Covid costs to go down, we need Covid to go down, but that means paying for it, i.e. to not have mask mandates, we need mask mandates. Politics then picks some entity to watch, i.e. new cases or hospitalisations, and adjusts the “Covid tax” to keep this value down. Overadjustment results in “dead weight”, but underadjustment does as well (because of the extra tax needed to get a wave back down) plus in dead people in addition to the economic dead weight.
Because Covid and public interventions aren’t well understood, this is really hard, and involves errors.
From Andrew’s 2011 blog post:
Now that we’ve seen a “taboo-busting entrepreneur” as POTUS who thought he knew better than everybody else, what lessons did we learn?
Frank Herbert (of “Dune” fame) invented the Bureau of Sabotage, because he felt government’s tendency to slow progress down was, on the whole, a good thing. So, holistically, maybe both entrepreneurs and bureaucrats are good if balanced against each other, and bad if not?
[Good is entrepreneurs, bad is bureaucrats. At some point this breaks down.
I recommend Mariana Mazzucato* as an antidote to the tired “private sector good -government intervention bad” mantra.
*e.g. M. Mazzucato “The value of everything; Making and taking in the global economy” Penguin 2018 (in UK)