We associate NPR with uncritical promotion of junk science (etc etc etc) and the scientist-as-hero narrative (which I absolutely hate).
So we should give them credit when they do the right thing and introduce some skepticism into their reporting. A couple people pointed me to NPR story, Fabricated data in research about honesty. You can’t make this stuff up. Or, can you?, which featured these bits:
GUO: The company told us in a statement that they’d pulled the original data set they sent to Dan Ariely, and it looked dramatically different from the published data from Ariely’s experiment. The company said they’d only given Ariely data for about 3,700 insurance policies. But in Ariely’s paper, he claimed he had data for more than 13,000 policies. That’s a difference of almost 10,000.
FOUNTAIN: In their statement, the company told us, quote, “though some of the data in the published study data is originally sourced from our data, it is clear that the data was manipulated inappropriately and supplemented by synthesized or fabricated data.”
GUO: The company basically confirmed everything that Uri and the Data Colada people had said about Ariely’s numbers. And the company made it clear that whatever had gone wrong with the data set had gone wrong after they had already given it to Ariely.
FOUNTAIN: We shared parts of the insurance company statement with Ariely, and he responded in an emailed statement, quote, “as I said two years ago, I was responsible for the relationship with the insurance company that provided the data for the paper. I got the data file from the insurance company in about 2007, and I can’t tell now who had access to it. Getting the data file was the extent of my involvement with the data.”
FOUNTAIN: But Michael says the problem with all these fake studies is bigger than the feelings of a bunch of academics like him. In Guatemala, there was this big investment. Hundreds of thousands of people were put through this experiment that was never going to work.
SANDERS: All of that time and money could have been spent on something different, a different intervention with a better chance of working if we’d known, right? This is not a petty, academic squabble about he said, she said. This is – like, this has real impact in the real world. So I’m pissed off about that. I’m also pissed off by how stupid it is. So when you see, like, eminent Harvard professor has committed research fraud, what you want is really sophisticated, clever ways of cheating, not, oh, I thought I’d copy and paste these observations ’cause they were higher, and that’ll make my result come together. This is dumb. And that annoys me. But it’s also a source of terror because that means there’ll be people out there who have cheated in a really sophisticated way who we haven’t caught and who we may never catch.
I agree with my correspondent that it’s good that they discuss the fraud and also the consequences of it.
Just one thing . . .
Two things, actually.
First, it’s great to see NPR doing some critical reporting on science hype. I’d also like to see them go back and address their past credulity in this area.
I think that one reason for the spectacular success of junk science in recent decades has been its promotion and endorsement by trusted news outlets such as NPR. So, once they realize they’ve been snookered, it would be good for them to address that: to forthrightly say in their story that, yes, they, NPR, had been promoting this junk science for years, and to consider what that implies about their science reporting more generally.
We can learn a lot from our mistakes, but only if we first face up to them.
Second, yes, science fraud is horrible, but there’s tons and tons of junk science that’s not fraud, it’s just bad work, in the sense of making strong claims that are not supported by the data. Himmicanes, air rage, ages ending in 9, etc etc etc. Some of this work (for example, the claim that beautiful parents are 36% more likely to have girls, or the claim that women were 20% were more likely to support Barack Obama during certain times of the month) were ridiculous on their face; other claims were internally inconsistent (for example, the paper with the voodoo dolls that described a three-day study as “long-term”); others had some surface plausibility but disintegrated under careful investigation.
My point is: Yes, let’s shine the light on science fraud. But don’t think that, just cos some highly promoted bit research is not fraudulent, that it should be taken at face value.
You’ve taken the first step, NPR. Don’t stop there!
Agree completely – and I think it is one thing, not two. What is missing is any responsibility taken by NPR. It is as if they are just reporters, and if people lie or cheat, their job is just to report what they say. And if they are caught, then report that. Where is NPR’s responsibility for failing to vet their stories?
Of course it is asking a lot for NPR reporters to do the kind of investigation that Data Colada did. Perhaps they are only obligated to report what researchers claim and then report what other researchers are able to uncover subsequently. We’ve seen this play out in political reporting as well. The problem with this passing the buck is that the buck never stops anywhere. I could use the same logic to justify citing published research as if it is correct, at least until somebody publishes a contrary finding. It’s not my responsibility to attest to the accuracy of the studies I cite – buck successfully passed. I think we all do this to some extent, but I’d note that in a legal setting (regulatory proceedings, from my experience), you do not escape responsibility in this way. If you cite things that turn out to be wrong, it can and does (perhaps not as frequently as it should) come back to haunt you. Reputation ultimately matters in some places, although not frequently enough in academia. In the press, I do think reputation matters (Fox and CNN being two examples, at different ends of the spectrum, of establishing a reputation of unreliability). However, we now see media reputations dismissed not only because of failure to vet their sources, but solely because of their political leanings. Once again, truth takes a back seat to feelings and beliefs.
I started out in journalism before meandering my way into academia. It was obvious to me at the outset that journalists *must* have a critical-reader level of expertise in any field they report on. They don’t need to be able to do the primary research themselves, but if there are competing narratives they need to first be able to recognize this is the case and second have the skills to identify the sources of disagreement so they can be reported. He said, she said is *never* good enough about anything. You’ve got to ask why your sources tell you what they tell you and get to the bottom of why you’re hearing different things. Very few journalists have these skills, including in specialized beats like business and science.
I would think that a good entry-level grounding in statistics, for instance, should be mandatory for science journalists. They should be able to read competing studies and figure out whether the issue is different samples (or sampling), measurement choices, modeling choices, etc. And then explain this to general readers in a nontechnical way. And of course they need basic subject knowledge in the fields they cover to know who to talk to get a representative set of views.
I never went to J-school, but from what I hear, this approach is trickling in, but slowly.
Peter:
Several years ago, when speaking at a conference of science journalists, I told the story of one of the embarrassing mistakes made by the Freakonomics team, where they reported, with complete lack of skepticism, a ridiculous claim based on the shakiest of statistical evidence. I made the point that Steven Levitt is one of the most qualified people in world when it comes to evaluating quantitative claims in social science—so if he could screw things up so badly, what is the hope for a journalist without Levitt’s training and experience? I concluded that for a journalist it’s important to get an outside perspective on a claim, rather than just accepting journal articles at face value. That said, sometimes the bad claims are easy to see through, and journalists don’t even try, perhaps because they’re intimidated by the journal publication.
Peter –
They should be able to read competing studies and figure out whether the issue is different samples (or sampling), measurement choices, modeling choices, etc. And then explain this to general readers in a nontechnical way.
That seems like a pretty tell order to me. With that level of expertise, I would imagine most would go into actual research rather then journalism. What you’re describing is what I would hope to see in a literature review or background section of a research paper. I’m not sure how well people can give a a context-relevant overview of differing viewpoints or contrasting findings without also being able to do the primary research. I think it’s a big problem these days that a LOT of people think that because they have an ability to understand basic principles of statistics they don’t need to have domain-relevant expertise to actually judge the context of the different views on different statistical findings.
I think it’s reasonable enough to expect a journalist to interview enough independent researchers who can provide that level of analysis to describe the reasons for differing viewpoints. That’s a step up from just “he said she said.” What’s clearly not OK is for a journalist to just interview a study’s author and leave it at that.
It is a tall order. So is expecting researchers to understand these things and communicate them clearly. Statistics is hard. I don’t think there are any easy or cheap solutions. Somehow, we expect a complex world to be simply understood – or, what is worse, we entrust “experts” since it is too complex for most of us. The next inevitable step is to trust AI (just watch those Apple commercials touting Apple Intelligence – which will probably do a better job than most journalists).
Fair enough. I certainly don’t think a journalist has to be an independent evaluator. But there are two things that need to be covered. One is to not only get independent opinions on a piece of research but to know which sort of reactions (from which camps if there are camps) are most important. The second is to be able to translate to general readers where the points of disagreement lie. I guess what I’m trying to say is that, from the perspective of science as a process, what counts is not just what one paper or research team comes up with but also how disputes and uncertainties are evolving. I don’t mean to say that journalists themselves should be arbiters, but they should know enough to place individual pieces of work in context. I think we all agree that most current journalism is way below this standard.
You got what you asked for. Journalists have been brow beat into parroting scientists by the scientists themselves. The Climate Disaster Lobby has relentlessly demanded that journalists never report any conflicting claims and stick to The Science(tm) as reported by The Highly Qualified Above Any Question Scientists.
You got what you wanted. Right?
If that’s directed at me? No, even if what you describe were true, (I don’t think it is), it’s not what I would remotely want.
What’s interesting to me is that your thinking is distorted enough to be so confident that’s something I’d want.
“I’d also like to see them go back and address their past credulity in this area.”
Journalists rarely do this for even non-science reporting, unfortunately.
It’s worthwhile to click the link to the NPR story and scroll down to The Hartford letter. They do an analysis of the disputed data that is devestating. It pains me that they use data as a singular noun.
Here’s a quote:
The data tells a different story: While the published data supports Ariely’s hypothesis, our data does not. The change in odometer readings in our data reflect what you would expect to see from real-life driving experience, illustrated by a bell-shaped graph, compared with the published study data, which shows a uniform distribution, as noted by DataColada.
Bob76
Bob:
I’m with ya on “data,” but I know that lots of people will argue the other way.
You called?
“Data”, like other mass nouns*, must be used with a singular verb, as anyone with any linguistic sense or a CS degree will tell you.
Heck, even the Microsoft Manual of Style says so.
(It says not to use either “datum” or “data are’, since people will see you as pretentious. (!!))
Actually, the Wiki article accepts the plural usage _in special cases_, but for “educated everyday usage” outside the safety of your clique, it’s a singluar mass noun. So NPR, not being in one of them linguistically challenged cliques, is correct in their usage.
My point being that I’m _incorrect_ to yell at you for the use of this abomination in your academic papers (although I take pleasure in doing so), but you are incorrect to be disturbed by the linguisticaly correct usage being used outside the context of one of those perverse fields that uses this abomination.
*: “The word data is most often used as a singular collective mass noun in educated everyday usage” Wikipedia.
David:
I don’t think I’m “incorrect” to be disturbed about that use. Being disturbed is just how I feel; it’s neither correct nor incorrect. I recognize that it’s not linguistically wrong for someone to use “data” as a singular; it’s still jarring to me.
The Guardian’s style guide says the following: data
takes a singular verb (like agenda), though strictly a plural; you come across datum, the singular of data, about as often as you hear about an agendum.
“data” can correctly refer to the set. Sets are things, so if you have one set, it is singular. The British seem to think all sets are plural, i.e., that the set isn’t a thing.
Wouldn’t that, then, be a dataset?
I have no problem with your saying “dataset”, if it makes you feel better.
It’s just that Brits use plural verbs for collective nouns routinely. “Queen were a band ….”
Not sure Kyle – maybe it depends on the tense and what the true subject of the sentence is. For example (I’m a Brit) I would say:
“Man United were pretty good against Everton last weekend” but also “Man United is still one of the better teams in the Premiership” but I’d struggle deciding between “Man United are lying 14th in the table” and “Man United is lying 14th in the table” -either would sound OK to me.
(I feel like we’ve had this converstation before btw)
Yes, Brits routinely don’t think of a set as a thing (at least grammatically). Not sure what British mathematicians write. Do they write “R is a complete ordered field” or “R are a complete ordered field”? They say “maths”.
The problem isn’t that they lack statistical training (although I’m sure that most don’t have any such training).
The problem is motivated reasoning. They want to believe this stuff, so why bother to subject it to scrutiny?
It’s a problem that goes well beyond scientific reporting. Indeed, it affects all reporting.
I could list all the examples from the last 2-3 years, but what’s the point? Everyone knows these examples, and whether you choose to believe them or not, or attach any importance to them, all depends on which tribe you affiliate with.
So three cheers for Planet Money — and no cheers for the standard repertorial lineup of “Morning Edition” and “All Things Considered” news staff, who will continue to promote BS.
Journalism jobs are still relatively plum positions (even with the death of old media) with a lot more applicants than spots. If the newspaper and TV/radio types put a higher priority on critical thinking, they could hire shrewder people to cover this stuff. Not just English majors from Vassar.
They made their bed. They can deal with the blowback. As it is, they really don’t get that much and just merrily move on to the next clip. So…just don’t have Gell-Mann amnesia.
Anon:
I think you got the economics of this one backward. Journalism jobs, with rare exceptions, are pretty bad. The journalism industry has been in a state of collapse for decades. The fact that there are a lot more applicants than spots is a bad thing, not a good thing! Journalists don’t have a lot of power, because (again, with rare exceptions), they can be easily replaced. When you have a field with many more spots than applicants, these are the plum positions!
My point is since there are a lot more people going after the jobs (even if they suck), the employers could easily staff them alternately. It’s just not in the English major culture to do so. Nor is the blowback sufficient to require them to do so.
Heck, given the overproduction of science graduates, it’s not that hard to just get someone from that pipeline. Sure, one might need to get someone with native English ability. And/or even (gasp) train the new hire a little in writing. But it’s not a situation where it’s impossible to find people. Maybe not computer science or chemE grads (since they still have good industry options). But there’s a lot of physicists looking for a gig. (There are VERY few of them getting the Goldman quant jobs you hear about.)
I’m not per se saying they should. Probably getting a good writer is more economically beneficial than a critical thinker. And who really cares if NPR gets some stories wrong? The midwits will lap it all up anyways.