Posted on

The Reasoning and Methodology Behind The Chart

 

tl;dr: There are lots of reasons. Many are subjective. More data would make it better. I am not a media expert.

Since my News Quality graphic got widely shared, I have been asked what my inspiration, methodology, and process was for creating it. I note that I have been asked this question by academics, journalists, and laypersons that care about accuracy and quality. Unfortunately, a lot of people don’t care about accuracy and quality. And a lot of those same people don’t like to read.

Why I Created It

I am frustrated by the reality that people don’t like to read. I LOVE to read and write. I have an English degree and a law degree, and I read and write every day for work. As a hobby, I read the great articles that are out there on the topic of media bias and accuracy. All of you who are reading this know that there is an abundance of great journalism out there—truly more than ever. I have the pleasure and privilege of reading a lot of this stuff, as do you.

But I know that the medium of a well-written article just doesn’t reach people who don’t read long things. In this post, I refer to such people as “non-readers” or “infrequent readers.” I am fully aware that the website MediaBiasFactCheck, and the organization Pew Research, and media research departments at many universities have large sets of empirical data available to review, and that those sources are more reputable than *just me*. But non/infrequent readers don’t read those sources. What do they read? Memes, which are often just two juxtaposed pictures with a pithy, terrible, one-sentence argument placed on top in large white letters. Tweets in which arguments are limited to 140 characters. They also prefer to watch videos, like YouTube “documentaries,” no matter how deceptively edited or spun.

Memes and tweets and YouTube videos spread quickly. They don’t take any effort to read, and people are convinced by them. They base their viewpoints upon them IN PLACE of basing their opinions upon long written pieces. To the extent that infrequent readers read, they prefer short articles that confirm their biases. Because they read very little, their comprehension skills and ability to distinguish good writing from bad writing is low. This is true for infrequent readers across the political spectrum. All of this is extremely disturbing to me.

Many non/infrequent-readers prefer easily digestible, visual information. I wanted to take the landscape of news sources that I was highly familiar with and put it into an easily digestible, visual format. I wanted it to be easily shareable, and more substantive than a meme, but less substantive than an article. I cite the fact that it has been shared over 20,000 times on Facebook (that I know of) and viewed 3 million times on Imgur as evidence that I accomplished the goal of it being shareable. In contrast, maybe one-one millionth the amount of people will read this boring-ass article about my methodology behind it.

Many non/infrequent readers are quite bad at distinguishing between decent news sources and terrible news sources. I wanted to make this chart in the hopes that if non/infrequent readers saw it, they could use it to avoid trash. For those of you who can discern between the partisan leanings of The Economist and the Wall Street Journal, I have to say this chart was not primarily made for your benefit. You are already good at reading and distinguishing news sources.

The fact that the chart is shareable does not necessarily make it TRUE. Having heard feedback from all corners of the internet, I know that many people disagree with my placements of news sources upon it. However, even people who disagree with the placements find the taxonomy helpful, because it provides a baseline for a discussion about media sources, which are inherently difficult to classify. Often, verbal and written discussions about news sources are limited to descriptions of sources as “good” and “bad,”  and “biased” and “unbiased.” This chart allows for a few more dimensions to the conversation. However, as discussed below, there are many metrics on which to evaluate and classify media, and this chart doesn’t include them all.

In creating the chart, I had to make (mostly) subjective decisions regarding four particular aspects, explained below.

Choosing the Vertical Categories

First, I considered what makes a news source generally “high quality” or “low quality.” “Quality” itself is an incredibly subjective metric. I figured a good middle category to start with would be journalism that regularly meets recognized ethics standards the profession, such as those set by the Society of Professional Journalists. http://www.spj.org/ethicscode.asp. Above and beyond that, I determined that factors that can make a particular article or broadcast “higher quality” include 1) a high level of detail, 2) the presence of analysis, and 3) a discussion of implications and/or complexity. So I created the categories of “Analytical” for sources that have 1) detail and 2) analysis, and “Complex” for sources that regularly have the discussions of 3) implications and/or complexity. To read the “Complex” and “Analytical” sources, you often have to be familiar with facts learned from sources ranked lower on the vertical axis. However, complexity is not always a good thing. Sometimes, real issues get obscured with complex writing.

Then, I considered what makes a news source “lower quality.” One of the factors is simplicity. Simplicity CAN lead to “low quality” if a deep issue is only covered at a very surface level. Simplicity is fine for stories like “a man robbed a liquor store,” but it’s often bad for, say, coverage of a complex bill being considered by your state legislature. There are sources that cover complex stories (e.g, Hillary e-mail stories, Trump foundation stories, and really, most political stories) in a VERY simple format, and I think that decreases civic literacy. Therefore, I created a below-average quality category called “Basic AF.” However, simplicity is not necessarily a bad thing. Sometimes you need “just the story.”

I have strong feelings about what factors really lower the quality of a source, and those are 1) sensationalism and 2) self-promotion in the form of “clickbait” headlines. Sources that engage in these actions are often geared toward attracting the attention of the non/infrequent reader. Sensationalism plays upon the worst emotions in us, such as fear and anger. Clickbait online articles have headlines that are rife with hyperbole. Then, the content of the articles themselves are loaded with adjectives (e.g., “clearly,” “obviously,” “desperately,” “amazing,” “terrific”) that are hallmarks of poor persuasive writing. That category definitely went at the bottom.

Few people quibble with the vertical categories as I have selected them, but as stated above, “complex” is not necessarily good and “basic” is not necessarily bad. Therefore, the “journalistic quality” arrow does not correlate perfectly with the vertical categories, and as a result, I myself find it to be an imperfect way to rank journalistic quality. However, they correlate enough that the ranking still makes sense, minus a few outliers. In particular, USA Today and CNN get pretty harsh vertical rankings due to my categories. I think USA Today is a pretty high quality publication, even though most of its stories are basic.

Note that the vertical categories do not take into consideration the presence of “truth” in a source. For example, the Wall Street Journal near the top, and CNN near the bottom, both generally report on things that are “true.” The vertical categories also do not differentiate between whether sources are more fact or opinion based. For example, both The National Review (near the top) and The Blaze (at the bottom) write very opinionated pieces.

 

Choosing the Horizontal Categories

Sorting sources based on partisan bias was a bit more straightforward, but I wanted to differentiate between the level of partisan bias. The categories are fairly self-explanatory. They are also the most highly debatable. Good arguments can be made as to whether a source is minimally partisan, “skews” partisan, or is “hyper” partisan. The “Utter Garbage/Conspiracy Theories” category is for those sources that “report” things that are demonstrably false and for which no apology or retraction is issued in the wake of publishing such a false story. These stories may include, for example, how the Obamas’ children were stolen from another family (on the right), or that the government is purposely poisoning us and changing the weather with chemtrails from airplanes (on the left). For the most part, even the “hyper-partisan” sites try to base their stories on truth (e.g., Occupy Democrats, Red State), and are held to account if they publish something demonstrably false. Generally, the closer a source is to the middle on this chart, the more they are taken to task by their peers for publishing or reporting something false.

The categorization of a source in the hyper-partisan or even utter garbage category does not mean that every story published there is false. Many articles may just be very opinionated versions of the truth, or half-truths. And occasionally, sometimes a hyper-partisan or garbage site will stumble upon an actual scoop, due to their willingness to publish stories that haven’t been sourced or verified. Their classification in these categories is mainly because they are widely recognized by other journalists as regularly falling short of standard journalism ethics and practices.

Lots of people have a problem with the category of “mainstream/minimally partisan.” To clarify, the category is called “minimally partisan,” not “non-partisan.” Because journalists are human people, they have opinions, and these opinions can make their way into their reporting. However, they also have professional standards and are held to account by their peers. Further, one can police one’s own biases to a certain extent if one is cognizant of them. The difference between “minimally partisan” and “skews partisan” is easily distinguishable by the intent of the organization. If they mean to be objective, that counts as minimally partisan here. If they mean to present a progressive point of view (MSNBC), or mean to present a conservative point of view (FOX News) that’s at least skewing partisan.

Choosing the News Sources to Include

The sources I initially chose include those I read most often and those I am exposed to most often through aggregators or other sources. They also include sources which I have reason to believe many others are exposed to most often. For people who get their news on the internet, their default browser home page is often a starting point for where to find news, and these home pages are often news aggregators. Yahoo, MSN, and the Microsoft Windows Edge Browser home page all present particular news sources. Many people also get their news sources from Facebook and Twitter (an alarming number, 40%, as I have seen in a recent survey, ONLY get their news from Facebook). Another aggregator is the Apple News App. Between these sources, I selected some of the most popular, making sure to include some in each category, and an approximately equal number of left and right partisan sources.

Note that I did not quantitatively determine how many sites are out there on each partisan side. Some people object to this and believe there are far more trash websites on one side or the other. I do not have the time or resources to conduct such a quantitative measure, so I did not conduct one. Some believe that because this measure is omitted, I am promoting a false equivalency between the sides. This may be true, if there is truly one partisan side that has significantly more garbage news sources. However, I believe there is value in presenting partisan balance within the chart so that more people across the spectrum are willing to take it seriously.

Many sources are not on here. That’s because there are hundreds of them. I could add twice as many easily, but then it would lose its readability. Remember, some people don’t like to read. For many, the words on the chart were too much.

 

Factors for Placing the News Sources on the Chart

I could have taken a number of empirical and quantitative approaches, but as stated earlier, but I did not set out to first conduct such a wide ranging study and then publish the results thereof. I just wanted to visually present a concept that many of us already hold in our heads. I am not affiliated with any research organizations that do this kind of work. I was actually very surprised that this chart was so widely shared, because  I am not an authority on this subject, and literally nothing I have ever written or drawn has attracted so much attention and scrutiny.

I am, however, experienced in defending my positions with facts and arguments, and I place value on the notion that assertions must be supported. I have outlined my support for these placements below.

One way to analyze sets of complex facts  is the approach used in our courts. There are some legal questions to which our courts have determined the best way to answer is through a multi-factor test. These multi-factor tests are appropriate for factual scenarios where there are many considerations to weigh. For example, in trademark law, to determine whether consumers are likely to be confused by competing trademarks, there is a 13- factor test. In patent law, to determine a reasonable amount of royalties to be paid for patent infringement, there is a 15-factor test. As a lawyer, I am comfortable with this multi-factor test approach, so I created one and applied it.

Given the popularity of this chart, though, I think it would be valuable to take my taxonomy and multi-factor test for placement and use it as a starting point for an actual study. A good empirical, data-driven study would probably look like a large panel of well-regarded journalists, writers, academics, and media observers poring over voluminous amounts of writing, spanning tens of thousands of articles and at least thousands of individual news sources, with the help of research assistants. It would probably use software to count and categorize words used in these articles and require cross-checking for verification of facts. As noted below in my list of factors, some just require a yes or no answer, but some are truly measurable and quantifiable. For each of the factors that are quantifiable here, I note that in my own evaluation, I only quantified these factors very generally, based on my observation and reading of headlines and articles. That is, I did not precisely count everything that could be measured. A real study could precisely quantify each of these factors, which would result in more precise placement of news sources. However, even in a quantitative study, certain aspects to placement will still be subjective; namely, the weight given to a particular factor in determining the ultimate ranking. It appears that any high-quality study of media sources requires both subjective and objective aspects, given that it is an analysis of written and spoken words.

Here are the factors I considered for each source, in no particular order. Below each factor is a note regarding what categories the factor weighted a source toward, and why. The notes also indicate whether a factor is quantifiable and could be more precisely measured in a future study for a future version of the chart

  1. Whether it exists in print

A “yes” answer weighted sources heavily toward “mainstream/minimal partisan bias” for several reasons. Print publication costs much more money, time, and effort to build than an internet one. Most print publications have significant numbers of staff members, including professional journalists. In order to have built a successful print publication, an organization will have had to spend time and effort building credibility among a significant audience. Reputation is necessary in order to have people buy newspapers for the purposes of getting the news. As a result of the above reasons, most print publications have longevity.

2.  Whether it exists on TV, and if so, whether it existed before cable

A “yes” answer weighted sources heavily toward “mainstream/minimal partisan bias” for similar reasons factor #1 (print). Cable lowered barriers to entry for radio broadcast news.

3. Whether it exists on radio, and if so, whether it existed before satellite radio

A “yes” answer weighted sources heavily toward “mainstream/minimal partisan bias” for similar reasons factors 1 (print) and 2 (TV). Satellite radio lowered barriers to entry for radio broadcast news.

4. Length of time established

Greater longevity weighted sources somewhat toward “mainstream/minimal partisan bias.” Longevity allows for the establishment of reputation (even a changing one) over time. However, newer sources can still be reputable and high-quality.

5. Readership/Viewership

This is a quantifiable factor. Greater readership and viewership weighted heavily toward “mainstream/minimal partisan bias” and somewhat toward the middle category of “meets high standards.”

6. Reputation for a partisan point of view among other news sources

“Reputation” is a highly subjective term, just like “quality.” Reputation varies and is fuzzy, but no one denies that it exists. Reputation testimony is admissible in court as evidence, so I included a few specific kinds of reputation as valid factors here. Other news sources talk about each other. If a large, established newspaper calls an internet website “left-wing,” or “right-wing,” and if these same internet websites call the large, established newspaper “the mainstream media,” they are in agreement as to each other’s partisan point of view.

7. Whether the source actively differentiates between opinion and reporting pieces

A “yes” answer weighted sources heavily toward “mainstream/minimal partisan bias” and was a determinative factor in whether the source was categorized at least in part as “mainstream” or fell completely into “skews partisan.” For example, the Washington Post, New York Times, and Wall Street Journal all have labeled opinion sections, while MSNBC, FOX, and Vox do not.

8. Proportion of opinion pieces to reporting pieces

This measure is also quantifiable. Greater percentages of reporting pieces weighted heavily toward “mainstream” and somewhat toward the middle category of “meets high standards.

9.Proportion of world news coverage to American political coverage

This measure is also quantifiable. Greater international news coverage weighted sources heavily upward. However, this measure is also subjective. I am of the opinion that if a source spends more time on world news, that indicates that it views itself as responsible for delivering all major news, rather than just focusing on ones that drive website traffic, like political gossip.

10. Repetition of same news stories

High repetition, in view of the medium, weighted sources heavily into the lowest vertical category for sensationalism. This was a main reason for CNN’s ranking toward sensationalism.

11. Reputation for a partisan point of view among my peers on social media

This factor sounds the most biased and subjective of all the factors, and it probably is. It is also typically the MAIN criteria upon which most people would rank these sources on the chart. There is some validity to using this measure; if your known conservative friend likes a source, it likely has a conservative point of view, and if your known liberal friend likes a source, it likely has a liberal point of view. There are obvious drawbacks to using this measure given the “echo chamber” nature of our social media feeds. If most of your friends have the same viewpoint as you, and you are all ideologically very partisan, then if they call a particular partisan source credible, that impacts one’s impartiality.

This factor was somewhat determinative of the placement of sources along the partisan spectrum, and hardly determinative of placement vertically.

12. Party affiliation of regular contributors/interviewees

This factor is also quantifiable. A balance of party affiliation weighted somewhat toward mainstream, and imbalance weighed to the partisan sides proportionally.

13. Presence of hyperbole in titles of articles

This factor is also quantifiable. The presence of hyperbole weighted heavily away from the center for partisanship, and weighted heavily downward for quality. I correlated more hyperbole with more partisanship and less quality.

14. Presence of adjectives in persuasive writing

This factor is also quantifiable. The presence of many adjectives weighted heavily away from the center for partisanship, and weighted heavily downward for quality. I correlated more adjectives with more partisanship and less quality.

15. Quality of grammar, spelling, punctuation, capitalization, and font size

Mistakes in grammar, spelling, punctuation weighted sources heavily downward for quality. Improper capitalization also weighted sources heavily downward for quality. Excessive capitalization (e.g., all caps) and excessive font size weighted heavily horizontally for partisanship and somewhat downward for quality. For example, the enormous, daily, all caps top headline on HuffPo pushed it well into the hyper-partisan category, but only down a little for quality.

16. Presence of an ideological reference or party affiliation in the title of the publication

Presence of reference or affiliation weighted sources heavily to the edges for partisanship and downward for quality (e.g., Occupy Democrats, Red State).

17. Effects of trying to actively control for my own known bias

I tried to evaluate my own bias and take it into account by first defining what my bias is and then making adjustments to correct for it. This exercise is difficult but crucial. It is imprecise and highly subjective. However, anyone who tries to make placements on this chart should engage in it.

I submit that a first way to evaluate your partisan bias is to categorize yourself on a number of political issues upon which there is consensus of what constitutes left, right, and center. Therefore, I started by evaluating my own views on what I think is “correct” and “true” on the issues of civil rights, taxes, business regulation, and the role of government in general. I am pretty adamant about civil rights and equality for all, especially for people of color, women, immigrants, and the LGBTQ community. I believe that places me in a somewhat left-of center category. On taxes and business regulation, I believe that neither “the government” nor “corporations” are all good or all bad. On the whole, I believe government does good things about 70-90% of the time and messes things up 10-30% of the time. I believe corporations do good things about 70-90% of the time and mess things up 10-30% of the time. As a result, I fall quite squarely in the middle, ideologically, on issues of taxes, business regulation, and the role of government.

In view of these evaluations, it would be fair to call me a left-leaning moderate.

To correct for this bias, I had to consider that there is a decent chance I am just wrong on what “the truth” or “the correct answer” is on one or more (or all) political issues. The likelihood that any one of us is completely right on all the issues is quite low. I have to acknowledge that there exists consensus about certain issues to the right of where I stand on them. That is, because approximately 46% of voters consider FOX News reputable and conservative principles acceptable, I cannot simply discount their likelihood of being right on the bet that I am right and they are wrong. As a result, I ranked Fox News higher on quality and less extreme on partisanship than I probably would have otherwise. I also ranked hyper-partisan left wing sites lower on the quality scale than I would have otherwise, and ranked complex/analytical conservative sources more centrally and higher than I would have otherwise.

Questions of bias, truth, and whether there is a center get philosophical and existential very quickly. All any of us can do is try to recognize and control for our biases.

Overall, this factor pushed conservative sources up and to the center, and liberal sources down and to the left in relation to where I might have ranked them purely on my ideological stances. It also pushed the sources into a relative balance that some argue does not exist.

A future study would benefit from having an somewhat equal number of left-leaning and right-leaning moderates arriving at a consensus to control for bias.

Factors Not Considered

I did not weigh the role of money from advertisers, ownership of sources, or corporate structure as factors in any meaningful way. I believe those factors are more closely related to the issue of media focus as opposed to media partisanship and journalistic quality. This chart was about partisanship and quality. It intersects with the topic of media focus only tangentially. I think the factors of money from advertisers, ownership of sources, and corporate structure can and do influence the topics that media sources focus upon.

Complaints about mainstream media focus are valid, but this is a whole complex topic in and of itself. Examples of these complaints include “why did it take so long to get mainstream coverage of the Standing Rock/Dakota Access Pipeline protests?” “Why did it take so long to get mainstream coverage about Bernie Sanders?” Why all the obsession with Hillary’s e-mails?” “Why the all-consuming coverage of all things Trump?” People point to money from advertisers, ownership of sources, and corporate structure as the root of these problems of misplaced focus, but I think it is more complex than that. Factors related to human psychology and attention, as well as modern technology likely play a role. Therefore, I left out the factors of money and corporations because it is an altogether different inquiry, and not necessary to resolve now in order to rank sources according to partisanship and quality. I believe factors 1-17 are sufficient to meaningfully place news sources along the continuum of this particular chart.

Edits, Arguments, and Future Versions

Based on thoughtful and legitimate feedback, I would likely make some edits on placement in my original chart. These include moving the Economist to the left of the midline, and splitting CNN into TV and Internet versions, and ranking the CNN Internet version in the middle circle while leaving the CNN TV version where it is. I would consider moving the Washington Post A LITTLE to the left, but I’d like to engage in a discussion about that.

I would be happy to have arguments about each of the listed factors above, and would entertain suggestions for other factors. I am also considering suggestions for future versions.

If others are inclined to take on the work of gathering data for the factors identified as quantifiable, I would be interested in supporting such work in some way.

Thanks for reading and thinking.

 

70 thoughts on “The Reasoning and Methodology Behind The Chart

  1. Thanks for this detailed account of your process. I wish such transparency were the norm. And thanks for the chart, too–brilliant work.

  2. Wow … this is the single best infographic/ mapping I’ve seen on this. Thanks for doing it.

  3. That was awesome. I just read it aloud to my husband and both of our kids overheard and came into the room to discuss your article (kids are 19 and 15). Thank you for the infographic and for this insightful account of your process. Even in places where I disagree with you, I can see how/why you made the decisions you did.

  4. I consider myself a morerate (I held my nose and voted for Hillary) and I disagree with a few of your classifications. IMO (and of my “peers”) CNN is definately left of center. The WPost and NYTimes are also both left of center, with the Times being less biased than the Post, and the Post being less biased than CNN.

    For example, the Post ran an article on the 10 questions that should be asked of Trump at the news conference to discuss Trump’s conflicts of interest as president. The questions were comprehensive, but the tone of the article highlighted the Post’s anatganistic attitude toward Trump. I could readily envision the same attitude coming from Sean Hannity, if he was given the opportunity to propose 10 questions to Hillary Clinton.

    Not a favorable comparison for the Post.

  5. This is great. Thank you for writing this and creating the info-graphic. Since my husband shared that info-graphic with me, I wanted to find out where it came from, and that search led me here.

    1. Thanks! I’m glad you enjoyed it.

  6. […] in the Denver area, took up this challenge, created a matrix by which to plot news sources, and has published her thinking behind the graphic, which has been viewed more than 3 million times in the last few […]

  7. Thank you for the well-constructed graphic. I really appreciate your explanation of your analysis, and I like that you’ve pointed out what you think are its strengths and weaknesses. Kudos to you on a job well-done!

  8. As an Edward Tufte/Visual Information geek, thanks so much both for the chart and your wonderfully detailed description of your metrics.

    You’re right – a lot of people don’t like to read, and reading from a computer screen makes it even worse. Your chart is an excellent way to grab attention and spur the conversation of what makes a news source a quality news source.

  9. Congratulations! Very, very well done. Wish I’d had this when I was teaching government.

    1. Thanks. You’ll be happy to know that many educators have reached out to me and are teaching this in their classes.

  10. That’s one long boring-assed explanation. 😉

    1. See, I told you!

  11. Thank you

  12. What a relief to find this chart! I’ve been so frustrated, especially during this election, with the wildly conflicting news sources and trying to distinguish (unsuccessfully) which may be the most unbiased nonpartisan sources. This article on reasoning and methodology is a cogent adjunct. Thank you!
    (I’m going to post the chart on my Facebook page but will add the link where I found both your comments and the reasoning and methodology article. I think they’re vital to a full appreciation of your work)

  13. I meant also say that I feel much better equipped to navigate the next four years! Thanks again!

  14. Great chart! Where would you put the New Yorker?

  15. This was a very illuminating breakdown, thank you for sharing! I found your chart via a friend’s FB like and followed it back to your Twitter. I see that some others have touched upon it there, but if you need a designer to update this infographic for you at any point, I’d love to offer my skills for use on this, for the sake of hopefully educating and reaching as many people as possible. Reach out to me if you’re interested, thanks!

  16. Cool! Numerical evidence would make this really interesting though. As an example, translating the y axis to sensationalist “factor”. Maybe based on the percentage of factual evidence in articles versus supporting opinion?

    1. I agree. I am contemplating how to do this.

  17. This was an absolute breath of fresh air for me. Thank you so much!

  18. I stumbled across this after sharing the graphic with as many people as I can for the last month. I have friends who are WAY right and WAY left and I was being inundated by the links of nonsense they share on FB. You have helped me filter out the extremes and focus on more analytical, complex and mainstream sources. Now I will share the link to this article and see if anyone reads it.

  19. Thanks for posting this as clarification! I don’t have much of a comment about your categories or placements (I’m imagining others have already given you more than enough feedback on this…).

    One thing I wanted to share though is how I conceptualize this problem as a fellow reader. In a sense I kind of mentally differentiate reputability of sources (per stuff like this) and likelihood of info being correct (which I think could be formalized by Bayes’ Rule). Basically, I try to assessing the quality of info rather than sources, but I take the sources into consideration. Unfortunately in my experience reputable sources aren’t always reliable. I’m thinking of stories like “one new study proves X weird thing”, out of context and pre-replication. Also, the Gell-man amnesia effect. Meanwhile, John Oliver who I’d put somewhere in the bottom left I thought did a good job in his bit on scientific studies. Even AP which I’d say is probably one of the most objective news sources isn’t immune to controversy. I could go on about errors from journals, textbooks, and experts and so on but I think you already understand that reputability of sources is a good heuristic to have but not a fail-safe.

    Here are some other things in my toolbox that I thing could benefit most people
    – Look up sources/authors on Wikipedia. They’ll usually have a section on bias. They’re not reputable, but they’re more reliable than they get credit for IMO. (Always feel free to double check them.)
    – Try to find primary sources or people with some credibility in the field or journalists with a good track record if possible.
    – Check against multiple sources and perspectives.
    – Mywot.com lets users rate websites’ trustworthiness.
    – Having good sources probably isn’t a replacement for critical thinking skills. Youtube has some intro-level playlists on this though, including lectures by college professors.
    https://www.youtube.com/results?sp=EgIQAw%253D%253D&q=introduction+to+critical+thinking

    Just my 2 cents on this. Good luck with all this! Congrats on the fame

    1. Also can’t edit, but to clarify for people reading this when I said “assessing the quality of info rather than sources, but I take the sources into consideration” I didn’t mean searching out opinions I already agree with and if a sketchy source says it back to me then that’s good enough….Definitely not! I meant trying to see if the balance of evidence favors their claims, seeking out differing perspectives, and credible authorities in that field in the light of critical thinking skills in order to fact check narratives and temper bias.

      Also there are some communities out there that are trying to help with this such as subreddits like r/politicalfactchecking, Facebook groups, and Quora. Would very much like to see these kinds of places get some more love

    2. Thanks for the comment. I have pondered several of the issues you mention. I agree that even good news sources publish articles that are outside journalistic norms, are poorly written, or are just wrong from time to time, which creates even more of a challenge. There are certainly a number of ways to categorize sources on top of the ways I have categorized them here. Unfortunately I think too many are missing even just the basic distinctions between high and low quality and left/right bias, which I think is why this chart resonated with so many.

  20. Why wasn’t National Review, Reason, or The New Republic on the chart?

  21. Shared this graphic on FB. Cited of course. I got one blow back “subjectivity” because of “lack of number scale”, and one about CNN’s position on the scale. I too am an analyst by trade, and saw Kyle’s post about numbers as well; I’m thinking about how one could quantify your assessment

    Acknowledge and appreciate the effort and attempt at “unbiased” assessment

    1. I definitely agree that quantifying the various criteria would be a worthwhile endeavor. Working on it. Thanks for the feedback.

  22. I have a question: Why was Politico not included on the cart? The Hill is there and the two occupy somewhat the same space in the news world.

    1. No reason in particular that it was left off. There are just so many news sources that even many reputable ones were left off just for space considerations. I agree with your assessment of Politico, though.

  23. This was really useful for me. I usually get my news from two delivered papers, and during the election I was seduced into Facebook coverage. I found the hyperbole and polarization from both sides very disturbing (and ugly). Then I discovered that the more I clicked on sites and shares that were interesting to me, the more “suggested posts” I received from Facebook. Before long, I hardly saw anything at all from the other side of the political spectrum (other than what I read in print). Add to that the fake partisan news on each side, and I think you’ve got a good explanation for how social media contributed to the extreme polarization of the electorate. I also have been contemplating how to judge different publications in terms of which can be trusted. Your chart is a very helpful start. Thanks so much.

  24. bookmarked!!, I love your website!

  25. I think that this resource is great (and I may use it in my critical thinking course). One correction I have is that you might want to rethink whether naturalnews.com counts as liberal. Though its title may sound that way, its content skews way right. This is from yesterday, for example: http://www.naturalnews.com/2017-01-24-goodbye-obama-you-sleeper-cell-traitor-and-enemy-of-america.html
    Their news feed shows itself to be pro-trump and extremely anti-liberal.
    There is a fairly common misconception that anti-vaxxers tend to be liberals, but in my experience this is not the case. In my experience (and I have a fair amount on this issue) anti-vaxxers tend to be libertarian types, while liberals tend to give the nod to science. There are, of course, exceptions to those tendencies, but just because naturalnews.com is anti-vax definitely does not make it liberal. Take a look there and I think you will agree that it should be moved to the other side of the chart.
    Overall, thanks for this great and useful graphic.

    1. I have gotten similar feedback before and agree. Based on the particular article, the source overall could arguably be placed on both sides. Thanks for the feedback!

  26. One thing that is missing from your analysis–I looked for it–was which stories the news sources chose to air (or not). IMHO, that would skew several of the sources further to the “biased” side (Fox News, I’m looking at you).

  27. A wonderful chart, thank you for sharing it, and I enjoyed reading about the process behind it. The level of thoughtfulness that went into it is a standard I wish more people appreciated, regardless of their political stripes. Finding common ground is a challenge.

    1. Thanks. I agree finding common ground is a challenge, but I think it is the most important challenge of our time. We need a coalition of the reasonable.

  28. Excellent explanation!!! Your chart rocks!

  29. Excellent methodology! Add me in as another academician who wanted a copy for my students. I’m a Graduate Teaching Assistant in Sociology, and I would love to share the image, as well as a link to your site so that they can see, evaluate, and understand your methodology with them next week when we go over research methods and critical thinking. 🙂 It’s always been critical, and more so now than ever I believe, that people (especially young people!) learn to think critically and learn to evaluate sources, and I think this is very helpful in doing so. Thank you for all your hard work on this!

  30. […] tech, and law. Great! But is she a legitimate source for creating this document? Here’s her post explaining why she created the document. So you see she did put a lot of thought into the […]

  31. […] Here’s the info on the chart and the criteria she used to make it by the author, Vanessa Otero: http://www.allgeneralizationsarefalse.com/?p=65 […]

  32. […] An exposition of the process used to create this brilliant diagram can be found here: http://www.allgeneralizationsarefalse.com/?p=65 […]

  33. […] of this graphic has an extremely detailed post about the reasoning and methodology behind it on her website3. Of course by nature there is a certain degree of subjectivity involved, so feel free to take with […]

  34. Thank you for offering this chart in an attempt to classify various news sources. Am I correct in interpreting your chart to imply that Fox News, though one must be careful to not “hang out there”, meets higher standards of journalistic quality than CNN? Can you clarify what you mean? Also, do you think there should be a distinction between written and TV news?

    1. I don’t necessarily think that FOX News meets higher standards of journalistic quality than CNN. In many ways I think their adherence to journalism standards is similar. As I discussed briefly in my post, the vertical axis is imperfect in ranking “quality.” Generally, higher on the chart means better quality, but not always. Because I placed sensationalism as lowest on the vertical axis, CNN got a pretty harsh placement. I do think there should be a distinction between written and TV news. I would actually rank CNN’s website and written news more squarely in the middle, but their TV news where it currently is. It is a challenge to correctly place entire sources on a two dimensional axis when there are many more than two factors that go into their overall placement. As a result, there are some incongruent placements, especially when comparing one source directly to another source.

      1. Thank you for your response. I personally find Fox news much more biased and sensational (and therefore having lower standards) in their approach to the news than CNN, which I observe attempts, at least ,to present both a liberal and conservative take on the issues they discuss. I view Fox news as clearly demonstrating their far right viewpoint and doing so in a manner that I view as argumentative and unprofessional.

  35. Thank you for this in-depth explanation! It is incredibly insightful. I shared this with you on Twitter but thought I’d put it here too. It seems like AllSides has done a lot of research into bias so you don’t have to!

    http://www.allsides.com/bias/bias-ratings

  36. What a lot of work! And this isn’t even how you earn a paycheck. But it is related to how you behave and the choices you make as an informed US citizen. Thank you!

  37. Excellent article, thank you for sharing.

    Another responder mentioned a publication’s selection of news stories. When trying to dig deeper on a story that feels biased to me, I’ll look up the rest of that writer’s stories for trending – it may help clarify their position by showing what subject matter they favor.

     That is likely too granular for an analysis about a news organization’s overall output, but does point to the question of whether or not it will tolerate [and print] a diversity of writer biases.

  38. Excellent post explaining your methodology. Really impressed with the thought you put into it. Have you thought about reaching out to Snopes to see if you could get more quantitative data on how often they end up correcting or refuting particular news sources?

    1. Thanks. I am working on making future versions based on quantitative data. It will be quite an undertaking, but I think there is a real need for it. I would definitely take into consideration numbers of errors and retractions and Snopes and Politifact would certainly be good resources on those numbers.

  39. Also, What about political endorsements?

    1. Those are certainly good indicators as well, but not completely determinative. They would influence the placement of sources like “local newspaper in liberal city” or “local newspaper in conservative city.” This past election, political endorsements as an indicator of left/right placement was anomalous, with the vast majority going to Hillary Clinton. I think there are reasons for that which I will write about in an upcoming post.

      1. Yeah, agreed on how anomalous this year was. I was thinking more a running weighted average over a long period of time kind of thing maybe.
        Very excited to follow your progress. Please let us know if you hear from someone who is doing research I’m this direction

  40. Thank you for your excellent chart and even more excellent explanation of methodology. One typo in #2 of your 17 points: the third last word should be TV, not radio. [Sorry, I get obsessive when I can’t parse a sentence easily.]

    I used to call myself a Rockefeller Republican (look it up if you have to) and miss the centrist days of before 1980 (Bill Clinton was close to an exception, but not completely)–not that everyone was centrist, but at least that was a respectable thing to be.

    1. Thanks for pointing out the typo. I’ll fix it. I think there is quite a bit of longing for a return to moderation and compromise. I hope that we are nearing the end of a cycle of polarization because so many of us want more civility.

  41. […] Because I care about methodology and accuracy, here is an explanation from the creator of her methodology for putting it together: http://www.allgeneralizationsarefalse.com/?p=65 […]

  42. […] chart has gotten a lot of attention lately, and its originator, Vanessa Otero, has explained all of the factors and assumptions that went into the placement of the various media outlets. […]

  43. Vanessa ~ I’m only about 50% through this post, and all I can write is: “Wow.” Excellent writing, clear voice, engaging detail/analysis. If I had to predict, I would bet that my students will **adore** this post, in that they’ll be privy to your process. SO thrilled that this will now be part of my curriculum — including the fact that I authorized its use with you prior to the instruction, thereby bringing home the importance of intellectual property. All best, m’lady! 🙂 Heather

  44. […] encourage you to read Vanessa’s “boring-ass article about her methodology.” As much as her graphic, I love the (4,400 word) description of her thought process, and that […]

  45. Given that the most recent approved reader post is from Feb 15, I’m surprised my comment from Jan. 23 is still awaiting moderation. I thought the links I supplied were germane, and would be of interest to you, and was looking forward to hearing your reaction to their content.

  46. I’m a PhD in Reading Education and am working to help older infrequent and nonreaders (high school and college students) understand how being such can adversely affect their lives. Therefore, I thoroughly appreciated, and enjoyed, your explanation in this post. I loved your infographic, but I immediately questioned your methodology and was excited to see a link to your thorough explanation. I wish I had time and resources to continue the research on this topic. In today’s climate, I think it is extremely important work. Thank you for taking time to share in such detail.

  47. Thank you for your effort and your detailed account of your process. I share similar concerns to you. I hope to see a similar and more detailed study carried out in the near future as more and more professionals consider the best ways to reach more people and end this debate over news and sources.

  48. I’m curious as to where The Intercept would figure into the chart.

  49. I appreciate the time you have taken to construct this and your description of your methodology. I do strongly disagree with your placement of The Nation, a high-quality magazine that has been around for a long time. Yes, it is a progressive, leftist magazine, but it is of high journalistic quality and it is of great journalistic value. You will read high-quality, investigative reporting in The Nation – and learn facts that may never appear in the more mainstream press, particularly around American foreign and domestic policy. The Nation does not belong on the line between “questionable journalistic value” and “but still reputable.” Again, thank you for your efforts.

  50. […] version indicates, the image was created by a patent lawyer named Vanessa Otero. As she admits in a post on her blog, Otero plotted the outlets in accordance with her own intuitions. This informality opened Otero up […]

  51. […] version indicates, the image was created by a patent lawyer named Vanessa Otero. As she admits in a post on her blog, Otero plotted the outlets in accordance with her own intuitions. This informality opened Otero up […]

  52. […] on media bias, but few of us know how she formulated the design.  Here’s the backstory:  http://www.allgeneralizationsarefalse.com/?p=65  Let’s use this info to be more mindful of the information sources we use to educate […]

Leave a Reply

Your email address will not be published. Required fields are marked *