I don’t know exactly how the amount of money affects the results, sorry. But certainly I do replicate the qualitative reflection effect (p < .001), and the quantitative percentage point shift is reasonably close to the original Tversky-Kahneman result (not identical).

]]>Decision (i). Choose between:

A. a sure gain of $240 [84 percent]

B . 25% chance to gain $1000, and

75% chance to gain nothing [ 16 percent]Decision (ii). Choose between:

C. a sure loss of $750 113 percent]

D. 75% chance to lose $1000, and

25% chance to lose nothing [87 percent]

Have you done this one on MTurk? If so, how do you implement the second scenario? Do you give them money first or somehow get them to give access to their account? How does the amount of money involved affect the results, is there some kind of curve people have figured out?

]]>it’s hard to explain why I can today successfully replicate Kahneman and Tversky’s effects from the 1970s in about 10 minutes.

Sorry, I am not familiar with what specifically you are referring to. If true, then these are some psychological/sociological laws and should be termed as such. Can you be more specific?

]]>It’s true that if one believes we live in a world in which in enormous “interaction” effects swamp all “main” effects, then the distinction I make is not so useful, but (1) that is because reliable/robust causal inferences are nearly impossible to systematically document if causality in the world hinges on complex interactions and cannot be parsimoniously described (a point the actual Andrew makes frequently), and (2) I don’t believe we live in such a world. If we did, it’s hard to explain why I can today successfully replicate Kahneman and Tversky’s effects from the 1970s in about 10 minutes.

]]>Also, if you read through the earlier thread I linked you will see that the omitted variable problem still exists for experimental studies if you want to extrapolate outside your “population” (which is almost always; see Deming’s “enumerative” vs “analytical” distinction linked to there). For example, my grandma is on at least a dozen different specific treatments at specific dosages, how well does any clinical trial approximate that? I’d guess no one has any idea.

]]>Yes, I agree with you, almost every paper without a (quasi-)experiment or an explicitly assumed formal model is making this same mistake. In my experience, many social scientists (especially, ironically, researchers who are primarily experimentalists) believe that as you add more and more control variables to an OLS regression of your DV on your (theoretical) “IV”, omitted variable bias “becomes less likely”. Of course, this is wrong. But that’s what they believe.

I think it has to do with the fact that it’s more difficult for researchers to think of additional alternative mechanisms / omitted variables if there are already many alternative mechanisms “controlled for” in OLS, so this creates the (false) perception that additional omitted variables are implausible or do not exist.

]]>Google says that about 25% of MLB players are left-handed, and the fraction of pitchers is similar, so assuming reasonable mixing, any given at-bat has around a 60-65% chance of same-handedness. J. Cross claims that pitchers are almost twice as likely to hit a batter with the same handedness. Given those numbers, there’s room for the interaction to look like pretty much anything.

]]>Some other ideas to prove that I’m giving this too much thought… At least in recent years, there are slightly more HBP on Sundays (the final game in a weekend series) and Sunday games are almost all days games and thus hotter. The day/night temperature difference might also be a reason why they would have been better off including season as a random effect rather insisting on a linear relationship between season and hbp rate. Stretches of seasons with higher hbp rates could also have more day games than we’d expect from a model implying a linear increase in the percentage of night games. Pitchers are also most likely to hit same-handed batters with inside fastballs and they might be somewhat less inclined to throw inside fastballs on hot days since home run rates increase with temperature.

]]>I really don’t understand how things got this bad when omitted variable bias, etc are well known to the stats community. Does anyone really believe the model presented in this paper approximates the correct one? Isn’t almost every paper making this same mistake?

]]>I’m not saying the paper in question is horrible; it’s just standard Psychological Science circa 2011: open-ended theory, lots of potential interactions, lots of arbitrariness in the model, key result is a p-value, lots of interpretation of the results. The number of researcher degrees of freedom is immense. Just for example, from the abstract: “Controlling for a number of other variables, we conducted analyses showing that the probability of a pitcher hitting a batter increases sharply at high temperatures when more of the pitcher’s teammates have been hit by the opposing team earlier in the game.” With a big pile of data and a flexible theory, you’ll have no problem finding statistically significant patterns in your data. The interpretation of the results in this depend a lot on the assumed additive and linear form for all the other predictors in the model, and it’s not at all clear why this additivity should make sense, given the authors’ own claim that a certain interaction is so important.

Again, I don’t see this paper as particularly bad; it’s just what people were publishing back in 2011, back before there was a general understanding of the issues of researcher degrees of freedom and forking paths. For yet another example, here’s footnote 7 of the paper:

Timmerman (2007) has shown that pitchers born in the U.S. South are more likely than others to retaliate when their teammates are hit by a pitch. We tested whether temperature remained a key variable after controlling for three measures of southernness: the location of the game, birthplace of the pitcher, and home location of each team. Only one southernness variable had a significant effect (see Table 2): Playing games in the southern United States increased the probability of a pitcher hitting a batter. This result suggests that a subculture difference (Nisbett & Cohen, 1996)—perhaps fan expectations—contributes to pitchers’ aggressiveness.

One could tell plausible stories like this forever.

]]>Regarding this specific paper, I suppose one could control for the importance of the game to the team. Teams often play games in hot weather in September as well as the months of high summer. The authors could examine whether playoff-contending teams are less likely to retaliate in September heatwaves than teams that are not contending.

]]>