This one pushes all my buttons

August Wartin writes:

Just wanted to make you aware of this ongoing discussion about an article in JPE:

It’s the same professor Lidbom that was involved in this discussion a few years ago (I believe you mentioned something about it on your blog).

Indeed, we blogged it here.

Here’s the abstract of Lidbom’s more recent article:

In this comment, I [Lidbom] revisit the question raised in Karadja and Prawitz (2019) concerning a causal relationship between mass emigration and long-run political outcomes. I find that their analysis fails to recognize that their independent variable of interest, emigration, is severely underreported since approximately 30% of all Swedish emigrants are missing from their data. As a result, their instrumental variable estimator is inconsistent due to nonclassical measurement error. Another important problem is that their instrument is unlikely to be conditionally exogenous due to insufficient control for confounders correlated with their weather-based instrument. Indeed, they fail to properly account for non-linearities in the effect of weather shocks and to control for unobserved heterogeneity at the weather station level. Correcting for the any of these problems reveals that there is no relationship between emigration and political outcomes.

This one pushes all my buttons:

– Breakdown of the traditional publication process, with a series of replies appearing in different places,

– Measurement error problems,

– The use of weather as an instrumental variable,

– Claims based on statistical significance.

I do not have the energy to look into the details here, but it seemed to be worth sharing.

10 thoughts on “This one pushes all my buttons

  1. regarding the freakanomics comment on the previous blog post:

    Recently I had happened to catch a Freakanomics segment on the radio. It was amusing. There was lots of discussion and arm waving people’s economic behavior under certain conditions. I can’t remember the exact situation, but it was supposedly amazing that people were making economic decisions that “left money on the table” and thus *chosing* not to maximize profit!!! Wow!!!

    Amazingly, however, there was a gigantic “but” at the end, which revealed that study was performed on a non-representative group of Ivy League undergrads; the amounts involved in their “economic” decisions were universally under $0.50; and the money was given to them, so it wasn’t even their own money. The narrator even suggested that people would probably behave differently if the money weren’t trivial, and even provided some discussion of how their behavior might change if the dollar values were much larger! All in all, I thought, a shockingly fair – even enlightening – discussion of the caveats.

    • Do you mean “enlightening” or “frightening?” Your recount of the story only leaves me wondering why such an experiment was ever reported on to begin with.

      • ha! Hundreds of experiments like that that have been reported. Dan Ariely. Yep, found him. Thats the dude who first introduced me to the we-gave-30-undergrads-ten-pretend-cents genre of behavioral economics. I believe the book was “Predictably Irrational”. He’s actually a psychologist.

        Confirmation. His wikipedia bio says “Ariely’s TED talks have been viewed over 15 million times.” ha ha ha, NYT best sellers and everything. Probably a master null hypothesis statistician. He’s checking all of Andrew’s boxes! :)

        Unlike the Freakanomics piece, Ariely provided no discussion whatsoever of the caveats of this research design. He plunged whole-heartedly into the amazing discoveries it yielded about human nature. There was a point when practitioners of this genre were claiming their experiments proved humans are by nature good. I’ve seen this type of work reported on NPR. I’m not sure if it still goes on, but in the last reports of this type of experiment that I read about, the stakes were much higher, presumably a response to the obvious criticism, though undergrads were still primary subjects.

        • Hi jim, you say:

          > Probably a master null hypothesis statistician

          I may be over-reacting here but it seems to me that on this and other pro-bayesian forums NHST is associated with bad science almost in a casual link. As if your science is bad *because* you do NHST.

          Experiments like “we-gave-30-undergrads-ten-pretend-cents” are bad regardless of what type of statistics you apply and I don’t see anything in NHST that makes them worst or in Bayesian statistics that could make them better.

          Often, however, I see a lurking assumption that if you go Bayesian then you will be honest about your results, knowledgable on the subject, and competent to translate your question into the Bayesian machinery (by no means a trivial task). If you stay frequentist instead you are bound, or more likely, to hack for p-values on ridiculous questions.

          It’s very reasonable to demand honesty and competence but if grant those qualities to the Bayesian camp then you should also ask what an equally competent frequentist would do in the same situation. My guess is that the difference between the two camps would be often small to negligible. Sure thing most of the bad science revolves around p-values. But I guess this is in part because frequentist stats is more popular and in part because Bayesian statistics is done by statisticians while frequentist statistics by people whose expertise is somewhere else.

          My concern here is that in this narrative (which may be all my mind of course) viable frequentist methods and results would be thrashed just because they are frequentist.

        • Andrew,

          Thanks a lot for replying. I’m aware of the papers you link and I’m also convinced that no serious statistician, no matter how radically Bayesian, would thrash a paper just because of p-values.

          However, in informal forums like blogs, not so much in papers, I still think that the message that a naive reader gets is that p-values are the problem and Bayesian is the solution. At least, that’s the message I got and it’s why I became interested in Bayesian statistics. Maybe it’s just me…

          Perhaps it’s a distortion similar to the statistics vs machine learning comparison. No serious machine learner would dismiss statistics, but there is some overhype around machine learning and an outsider is inclined to see statistics as somewhat simplistic and old-fashioned.

          (Admittedly, your last post “Bayesian methods and what they offer…” is a lot more balanced than what I’m picturing here!)

        • Dario, thanks for your comments.

          I recognize that these experiments are horrendous regardless of the analytical method employed – that’s why I pointed them out. In most of my comments on this blog I express concerns with fundamental flaws in the design of experiments and the numerous unlikely-to-be-satisfied assumptions implicit to those designs.

          Yet however horrendous they might be, the p-value paradigm lent credence to the results of this type of experiment, and – for a time, anyway – those results were influential. Thus my reference to p-values is to mock how easily the method is abused by a (speaking generously) credulous practitioner – and also as a joke for Andrew.

          I’m not a statistician and I don’t advocate for any statistical method. I understand can apply some frequentist methods, but I can’t tell you the first thing about how to apply Bayesian methods. And in all my comments here you wouldn’t find a single one that advocates for Bayesian methods.

          My view is that every question in science in every discipline should be approached with a variety of methods. The test of those methods isn’t whether we think they make sense or not. The test of methods is weather they actually work – whether they are validated and reproduceable. From that perspective, though I agree that it’s appropriate for some situations, the null hypothesis approach doesn’t seem to work very well. Even when no effort has been made to reproduce the results, it often yields results that are ridiculous or preposterous, that could only be accepted at face value by a strongly interested or shockingly credulous individual. So in that respect, I think it’s mostly bogus, but I’d hardly say Andrew lead me to that conclusion!! It’s trashy output is all over the media.

    • That freakonomics episode seems to have been the one on what questions people ask. This was done entirely by psychologists, and what they did would never pass muster in an econ journal. And the welfare statements would make established welfare economists pull their hair out. Just saying…. econ clearly has its problems, but we’re not nearly as naive as the researchers in that freakonomics episode

      • “That freakonomics episode seems to have been the one on what questions people ask. ”

        Yes I believe that’s the one, and yes, it’s not economists but psychologists, as per my discussion above. From my experience with that genre of work, the Freakanomics episode was startlingly thorough in covering the shortcomings, so if you think it’s bad OMG don’t look at Dan Ariely’s books! Brutal.

      • And I’ll bet that the economists’ GRE scores are higher than those of psychologists (per a recent discussion we had, based on the Marginal Revolution bashing of epidemiologists’ GRE scores) – so there!

Leave a Reply to Dario Cancel reply

Your email address will not be published. Required fields are marked *