Measuring media bias

Tim Groseclose and Jeffrey Milyo wrote a paper on “A measure of media bias.” Here’s the paper, and here’s the abstract to the paper:

We [Groseclose and Milyo] measure media bias by estimating ideological scores for several major media outlets. To compute this, we count the times that a particular media outlet cites
various think tanks and policy groups, then compare this with the times that members of Congress cite the same groups. Our results show a strong liberal bias: all of the news outlets we examine, except Fox News’ Special Report and the Washington Times, received scores to the left of the average member of Congress. Consistent with claims made by conservative critics, CBS Evening News and the New York Times received scores far to the left of center. The most centrist media outlets were PBS NewsHour, CNN’s Newsnight, and ABC’s Good Morning America; among print outlets, USAToday was closest to the center. All of our findings refer strictly to news content; that is, we exclude editorials, letters, and the like.

They fit a version of an ideal-point model to mentions of “think tanks and policy groups” by Congressmembers and media outlets and they find, basically, that most of the newspapers they look at quote a mixture of groups that is similar to moderate-to-conservative Democrats, most of the TV shows are comparable in quotations to conservative Democrats in Congress, and two of the more partisan Republican news organizations (Fox News and the Washington Times) have quotation patterns that are comparable to liberal Republicans in Congress.

This makes sense–as the authors note, surveys have found that many more journalists are Democrats than Republicans, but partisans of both sides have to moderate their views in order to maintain journalistic credibility.

I wonder to what extent these bias measures depend on the issues under consideration. There’s also the question of the relevance of quotation patterns to the larger questions of bias. As well as the question of what role the press should be expected to take in a representative democracy–for example, will a mass-readership press be expected to hold more left-wing views so as to be popular with readers, or more right-wing views so as to be popular with advertisers? There’s also the difference between local and national news. Lots to think about.

Brendan’s comments

P.S. Here are Brendan Nyhan’s thoughts criciticm of the paper. Brendan’s criticisms seem valid to me; notheless I’m a bit more positive than Brendan is about the paper, I think because the problem of studying media bias is tough, and I’m impressed about what Groseclose and Milyo did manage to do. Perhaps just my own bias in showing an affinity with quantitative researchers . . . I do agree, though, that “bias” isn’t quite the right word to discuss what Groseclose and Milyo measure, since “bias” implies a deviation from some unbiased position or truth, which I don’t see them measuring.


I also have a few comments on the presentation of the results:

Although Groseclose says on his webpage that the paper is scheduled to appear in a journal, perhaps it’s not too late for them to fix things a bit . . . First, I’d replace the tables by graphs. This would be really easy: for example, Table 1 could be kept exactly the same, except that the three columns of numbers could become columns of dots (scaled from 0 to 100 for the first column, 0 to 600 in the second column, 0 to 1400 in the third column. Or the second and third could be done on the log scale. Table 2 could also be a dotplot (just as Bill Cleveland wrote about in his 1985 book), etc etc. If making graphs is too much work, then they could at least round off those numbers like “56.9” to the nearest integer. (Just as SAT scores are reported as “560,” not “558.”)

In Tables A1 and A2, please please please don’t list the media outlets alphabetically! Howard Wainer and others have written on this. Use the same scale you used elsewhere, from most liberal to most conservative. (You should also do this with Table IV. “Most to least centrist” is confusing. Actually, you can combine Tables IV, A1, and A2 by plotting the estimate, with little horizontal lines showing the range of estimates under different models.

Figure 1 is ok. But the x-axis is weird. Labels every decade or two would be enough. Labeling every four years just makes it hard to follow.

Figure 2 really confused me for awhile. The way it’s written, it looks like the news organizations have a much wider range than they really do. If done right, there’s no need for all these ugly arrows. Play with the white space a little and you won’t need, for example, to have some news orgs take up two lines. Also, it looks a little weird that some of the Congressmembers are given full names and others just get first initials.

1 thought on “Measuring media bias

  1. Too bad your endorsement of Nyhan's critique is so tepid. It is even more puzzling given that you realize that "bias" is a misnomer for what G&M are measuring. Such terminology makes their paper positively misleading. And there are flaws in their reasoning that Nyhan hasn't even addressed; for example, even if the U.S. population as a whole were (say, by definition) unbiased, there is a big leap from measuring the median position of the members of Congress to asserting that it equals the median position of voting population to equating that to the median position of the whole population.

Comments are closed.