Measuring dynamic media bias

Our primary measure of media bias is created by weighting a channel-day’s DIME campaign finance (CF) score based on how much time an individual is on the screen.^¶ For example, if an individual is on the screen for 100 s and has a CF score of 0.5, this results in a weighted score of 50. We aggregate up these weighted scores per channel day and then take the weekly average per channel. To interrogate the validity of this measure, we provide basic descriptive patterns below. First, the measure appears to have face validity: the raw densities of the weighted CF score per channel, displaying remarkable similarity between CNN and MSNBC (weighted CF score of –9.7 and –14.1, respectively), and a much more right-leaning Fox News (weighted CF score of 49.8). Fig. 1 displays the mean level of the weighted CF scores for the top five programs per channel.^ǁ Programs on Fox News are on average more conservative than programs on CNN and MSNBC. Our program-level measures reveal, at least based on the guests that appear on the show, popular primetime news shows on CNN—such as Anderson Cooper 360 or CNN Tonight—are more left leaning than well-known evening shows on MSNBC such as The Rachel Maddow Show or The 11th Hour with Brian Williams.

Because our measure of media slant relies heavily on the DIME estimation process, interrogating the validity of this measure requires demonstrating that such media bias is “visible”; we need to show whether the utterances of a speaker with a high CF score, when read with plain eyes, are deemed more conservative than those of a speaker with a low CF score. We employed two research assistants to sample 978 news transcripts and broke them up by speaker into 19,582 blocks of text. We then assessed whether a block of text demonstrates a liberal, neutral, or conservative slant. To prevent the possibility that research assistants’ prior knowledge about partisan leaning of certain actors biases the coding results, we anonymized all actors from the transcripts. The resulting exercise generated moderately high intercoder reliability, with weighted Cohen’s Kappa values of 0.664, 0.55, and 0.66 for CNN, MSNBC, and Fox News, respectively. The correlation between our primary measure of media bias and the human coded estimate was 0.72 (see SI Appendix for details).** Based on these correlations, we argue that our main measure is qualitatively better, or at least comparable, to the measures based on text-scaling–based methods adopted in previous literature based on face validity and construct validity. For instance, Gentzkow and Shapiro (8) report a correlation of 0.40 between their text-scaling approach and a hand-coded measure of newspaper ideology. The correlation between our measure and a text-scaling measure (9) was 0.59. As an additional validation, we compare the weighted ideology scores of members of congress that appear both in Padgett et al. (10) and in our data.

The granularity of our measure allows us to offer a richer set of descriptive patterns. First, we can examine the dynamics of polarization by calculating the distance between the weekly means of media bias, comparing each channel with the other channels. As Fig. 2A shows, while the distance between CNN and MSNBC has remained mostly unchanged over time, the relationship between Fox News and the other channels has polarized—growing increasingly distant from one another. This pattern of polarization started a bit earlier between Fox News and CNN than between MSNBC and Fox News, although the polarization accelerated for MSNBC and Fox News starting in 2018. In Fig. 2B, we first plot the distribution of weighted CF scores by channel and over time. While Fox News is always to the right of CNN and MSNBC, CNN and MSNBC become more in sync over time. Between 2011 and 2015, CNN was consistently to the right of MSNBC, which is no longer the case after 2015. Additionally, the channels seem to mirror the public mood—the channels shift to the right, in parallel when Republicans gain power in 2016, but shift back to the left following the Republican defeat in the midterm elections. The correlations between these trends are fairly high ( $r_{\mathit{Fox},\mathit{CNN}}=0.68$; $r_{\mathit{Fox},\mathit{MSNBC}}=0.72$; $r_{\mathit{MSNBC},\mathit{CNN}}=0.68$). These results are similar even if we examine only scores that do not contain politicians (and, therefore, are not driven by prevalence of administration officials on television).

We also see interesting patterns by timeslot (afternoon, when most hard news is aired; morning, when talk shows appear, such as Morning Joe or Fox & Friends; and primetime, when popular shows air such as The Rachel Maddow Show, Hannity, and Tucker Carlson Tonight). Scores in the afternoon slot—when hard news shows air—are far more correlated between channels than in other time slots. While news shows across all time blocks have polarized since 2017, we find that this pattern is the most pronounced among primetime shows, with Fox News steadily becoming more conservative and MSNBC and CNN sharply becoming more liberal during the Trump presidency. The fact that morning shows on Fox News leaned more conservative than prime-time shows is consistent with the evidence that then President Trump was a regular and avid consumer of Fox & Friends. The granularity of our measure also allows us to see the variations in ideology in each of these programs over time. We find, for instance, that The Rachel Maddow Show became more liberal through the Trump presidency and that The O’Reilly Factor quickly turned farther right until the Fox News Channel expunged him from the channel (Fig. 3).