Disentangling the Influence of Recommender Attributes and News-Story Attributes: A Conjoint Experiment on Exposure and Sharing Decisions on Social Networking Sites

The Role of News-Story Attributes

A range of reasons explain why individuals choose or choose not to read or recommend a news article, such as the “shareworthiness” of a news story (Trilling, Tolochko, and Burscher Citation2017). We focus on two central news-story attributes that are particularly prominent predictors of news selection decisions in the selective exposure and sharing literature: attitude alignment and party favorability. We also include a third news-story attribute—theme—in our study to increase the realism of the news story description and account for masking effects.

When faced with a choice between reading or recommending information with which one disagrees or agrees, individuals tend to choose the latter over the former, exercising a confirmation bias toward attitude-consistent information (Hart et al. Citation2009; Stroud Citation2017; Lodge and Taber Citation2013). Such biased selection behavior is often explained by the theory of cognitive dissonance (Festinger Citation1954), that is, that people seek to reduce exposure to attitude-inconsistent information and increase exposure to attitude-consistent information. Although exposure and sharing decisions are conceptually distinct and have different causes (Liang Citation2018), people also tend to refrain from sharing content with which they disagree (Arendt, Steindl, and Kümpel Citation2016; Shin and Thorson Citation2017). When people do share information they disagree with, the objective is not necessarily to recommend the content but rather to provide nuances or criticize the shared content through commenting (Anspach Citation2017). However, in this article, we restrict our understanding of sharing behavior to individuals’ acts of recommending information.

Message alignment describes individuals’ preferences for attitude-consistent over attitude-inconsistent information. News headlines are often framed in such a way that one side of an issue is more prominent than another, and people often use such message-alignment information when they make news-selection decisions (see Knobloch-Westerwick and Meng Citation2009; Winter, Metzger, and Flanagin Citation2016). Based on the large body of evidence from the literature on selective exposure, we expect people to prefer to read recommended news that features information with which they predominantly agree rather than disagree. Turning to selective sharing, prior studies have also indicated that people tend to share like-minded content (Bigman et al. Citation2019; Liang Citation2018; Coppini et al. Citation2017). However, as argued by Bigman et al. (Citation2019), some might be reluctant to share a like-minded message for fear of social repercussions if they feel the message goes against the perceived prevailing sentiment of their SNS. Thus, while we also expect people to prefer to recommend news that features information with which they predominantly agree, we do not know whether the effects of attitude-alignment are stronger for the decision to recommend news than to read news. This leads us to formulate this research question:

Party favorability as a news-story attribute refers to systematic biases in people’s selection of news stories that favor a party or candidate that they like over one they dislike (e.g., Iyengar et al. Citation2008; Meffert and Gschwend Citation2012). Political news usually involves one or more politicians, party leaders, specific parties, or representatives from political parties, and a considerable body of literature (e.g., Bolsen, Druckman, and Cook Citation2014; Cohen Citation2003; Slothuus and De Vreese Citation2010; Iyengar and Westwood Citation2015) demonstrates that individuals use such information about parties to filter political information. A similar pattern has been found for people’s sharing behavior on SNSs (Shin and Thorson Citation2017; Barberá et al. Citation2015; An et al. Citation2014). This leads us to assume that people’s opinions about political parties in recommended news stories affect their likelihood of selecting such stories to recommend or read on SNSs. Thus, we assume that people are both more likely to read and recommend news that features a party they like rather dislike. However, we do not know whether the effects of party favorability are stronger for the decision to recommend news than to read it. This leads us to formulate this research question:

The Role of Recommender Attributes

While news-story attributes provide users with information about the content itself, individuals’ friends and acquaintances who share news on SNSs provide additional information that may aid in the selection decisions. Importantly, Kaiser, Keller, and Kleinen-von Königslöw (Citation2018) provide evidence to suggest that users who encounter shared and endorsed news on SNSs may form opinions about such shared content based on who the recommenders are and their relationship to the recommender. Users may also consider the recommenders’ intentions for sharing the information when they decide whether to read and/or recommend the shared content. We refer to such information as recommender attributes. Particularly, we focus on four different recommender attributes that prior literature has found to be influential on news-selection decisions on SNSs: the strength of the friendship tie with the recommender, the political knowledge of the recommender, the popularity of the recommender, and the recommender’s party preference. In addition to these attributes, we also study the effects of the recommender’s language skills, gender, and religious affiliation to avoid masking effects and to provide a more complete profile of a recommender to increase the realism for the respondents in our study. To be clear, our claim is not that these attributes constitute a complete set of characteristics of recommender profiles but rather that the attributes we include mirror a wide set of characteristics of recommenders on SNSs and in the academic literature.

Starting with research on the strength on SNSs, prior literature has demonstrated that the tie strength between the user and recommender is an important recommender attribute in terms of shaping users’ exposure decisions. People’s connections on SNSs, such as “friends” on Facebook, consist of a variety of different types of relationships, stretching from close friends and family to co-workers and distant relations. This means that SNS connections consist of both strong and weak ties (Granovetter Citation1977). While weak ties on SNSs have the potential to offer information that the user would otherwise not be likely to happen upon (Bakshy et al. Citation2012), strong ties seem to be more influential in terms of selection behavior on SNSs (Bond et al. Citation2012; Valenzuela, Correa, and Gil de Zuniga Citation2018). This is further suggested by evidence from an experiment on incidental news exposure on Facebook, showing that Facebook users are more likely to read articles recommended by strong ties than by weak ones (Kaiser, Keller, and Kleinen-von Königslöw Citation2018). While we assume that people prefer to both read and recommend news stories recommended by strong ties over those recommended by weak ties, we do not know whether there is a difference in effect sizes between the decision to recommend and the decision to read a news story. Consequently, we pose the following research question:

Building on the information utility approach (Atkin Citation1973), Kaiser, Keller, and Kleinen-von Königslöw (Citation2018) argue that people are more likely to read news that is recommended by someone who provides users with useful information and has relevant expertise (Metzger and Flanagin Citation2013). Several studies have also demonstrated that news recommendations from opinion leaders influence news selection behavior on SNSs (Turcotte et al. Citation2015; Anspach Citation2017). This means that attributes pertaining to a recommender’s expertise and utility, such as political knowledge and language skills, should influence selection behavior. We assume that people prefer to read and recommend news stories that are recommended by politically knowledgeable contacts. Thus, we ask the following research question:

The number of social endorsements from others on SNSs is another important attribute that several studies have found to serve as a heuristic cue that signals the utility of a recommended news story. If many users find a news story valuable enough to be endorsed, it is more likely that users will expect the shared news stories to be useful (Atkin Citation1973; Knobloch-Westerwick et al. Citation2005; Messing and Westwood Citation2014; Winter, Metzger, and Flanagin Citation2016). Such social endorsements have been found to increase the likelihood that the recommended news stories will be selected for further reading by users (Winter, Metzger, and Flanagin Citation2016; Messing and Westwood Citation2014). We assume that people prefer to read and recommend news stories recommended by contacts who are popular (i.e., operationalized as others often sharing their recommended content) on SNSs. Therefore, we ask the following research question:

Party favorability is not only an important attribute in terms of reading and recommending news stories that mention political parties but also in terms of the recommender’s party preference. For instance, Kaiser, Keller, and Kleinen-von Königslöw (Citation2018) provided evidence to suggest that political similarity between the recommender and the user—when the recommender was politically knowledgeable—increased the chance of exposure to recommended news. We assume that people prefer to read and recommend news stories recommended by a contact who prefers a party they like. Thus, we posit the following research question:

The Influence of Recommender Attributes on News-Story Attributes

So far, we have discussed how we separately expect news-story attributes and recommender attributes to influence news reading vs. recommending decisions. On online news sites, users are likely to come across news stories without encountering the news recommender attributes discussed above. However, on SNSs, users are likely to come across the identity of the news recommenders as well (e.g., news stories that are shared by friends and acquaintances on SNSs rather than by the Facebook page of an online news site). In other words, users are likely to be presented with both news-story attributes and recommender attributes simultaneously on SNSs. Do recommender attributes influence the effects of news-story attributes? A growing body of literature has explored the conditions under which people selectively expose themselves to or share news (e.g., Messing and Westwood Citation2014; Winter, Metzger, and Flanagin Citation2016). No prior study, however, has provided evidence on the extent to which recommender attributes influence the effects of news-story attributes.

Our theoretical argument for why information about the recommender could alter the effects of news-story attributes on exposure and sharing decisions builds on the assumption that a news story’s perceived utility can be a more influential attribute than the message alignment or party favorability of a news story (Atkin Citation1973; Messing and Westwood Citation2014)—at least for exposure decisions. In other words, knowledge about the recommender may include more decision-relevant information and hence override the effect of knowledge about the news stories. For instance, Messing and Westwood (Citation2014) provided evidence from two experiments showing that social endorsements (from unknown recommenders), operationalized as the number of recommendations a headline had received from politically heterogeneous individuals on SNSs, overrode selective exposure to like-minded news sources. That said, Winter, Metzger, and Flanagin (Citation2016) found no effects of the number of likes on selective exposure to like-minded news-story messages. Alternatively, knowledge about the recommender may alter the weight put on certain news-story attributes by implicitly increasing their information utility. For instance, receiving information that the recommender is politically knowledgeable could indicate that a political news story is likely to be useful (Kaiser, Keller, and Kleinen-von Königslöw Citation2018), regardless of the message’s political alignment or whether the featured party is favored.

Limited evidence exists regarding how recommender attributes (from known recommenders) influence exposure and sharing decisions as the literature on the effects of social endorsements on selective exposure has provided mixed results so far. Since we here include a range of recommender attributes rather than one or two, we formulate two research questions:

Data

The experiment (N = 3,860) was fielded in wave eight of the Norwegian Citizen Panel (NCP), which ran from May 11 to June 6, 2017. The NCP is a probability-based, nationally representative online survey panel. Participants were invited to two surveys a year and recruited in three waves of postal recruitments (25,000 invitees in waves 1 and 3; 22,000 in wave 8). The participants were randomly selected for recruitment from Norway’s National Registry—a list of all individuals who either are or have been residents of Norway, maintained by the official Tax Administration. This means that the entire Norwegian population had an equal and known probability of being invited. The recruitment rates for those invited were high—20%, 23%, and 20%, respectively. For more details about response rates or other methodological matters, we refer the reader to the NCP methodology report (Skjervheim, Høgestøl, and Bjørnebekk Citation2017). Our experiment was fielded to a random subsample of 60% of all respondents participating in survey wave 8 (which had 6,087 responses in total). Table 1 shows descriptive statistics of this subsample.

The Experimental Design: A Choice-Based Conjoint Experiment

Conjoint experiments are particularly well-suited for studying the causal effects of multiple independent variables and comparing the causal treatment effects of several attributes simultaneously (Knudsen and Johannesson Citation2019). While conjoint experiments have been used in different disciplines for decades, they have experienced a renaissance ever since Hainmueller, Hopkins, and Yamamoto (Citation2014) argued for their application to political science and causal analyses. Their popularity has also grown in communication science ever since their benefits for causal analyses were introduced to political communication research (Knudsen and Johannesson Citation2019).

We use a full factorial conjoint design—or what Hainmueller, Hopkins, and Yamamoto (Citation2014) describe as a typical choice-based conjoint design, where the stimulus consists of a two-column table showing two different profiles in which the rows describe different attributes of the profiles. The experimental design in this study is illustrated in Figure 1(a–d). We asked 3,860 Norwegian citizens to choose between two hypothetical social-media-network profiles. These profiles randomly varied regarding attributes that described the recommender (seven attributes; see Figure 1(a)) and attributes that described the recommenders’ 10 last shared news stories (three attributes; see Figure 1(b)). The recommender attributes were tie (strong/weak), gender, language skills, political knowledge, popularity on social media (whether many or few share the persons’ posts), religious affiliation, and the recommenders’ party preference. The news-story attributes were message alignment (counter-attitudinal, balanced, or pro-attitudinal), party favorability, and issue (five issues). The list of all possible attributes and values and the exact wording shown and explained to the respondents (translated from Norwegian) are displayed in Figure 1(f). Importantly, each attribute can take several values, and the two profiles are randomly generated. The order of attributes is randomized as well. For the recommender attributes, 1,296 possible profiles could be created. For the news-story attributes, 135 possible profiles could be created. While the possible profile combinations far exceed the profiles presented to our respondents, it is, as argued by Hainmueller, Hopkins, and Yamamoto (Citation2014), common practice in conjoint experiments to allow more possible profile combinations than there are observations in the data set, although only a small part of the possible profiles is ever observed. Hainmueller, Hopkins, and Yamamoto (Citation2014) demonstrate that we do not need to observe all possible combinations to assess the relative treatment effects of each value as the construction of possible combinations is completely randomized.

Figure 1. Overview of the Experimental Design.

Note: This figure illustrates the experimental design. The box on the left shows a mock-up in English of the experiment as seen by the respondents; a shows how information about the recommender was presented, b the news story, c shows the question wording for reading, d sharing, e shows how a–d were randomly assigned between subjects, and f shows all the possible attributes and attribute values.

Display full size

The effects of party favorability, for both the party in the news stories and recommender’s party preference, were measured by matching the party treatment(s) shown in the conjoint design (the table of profiles; see Figure 1) to measures of whether the respondents liked or disliked that party. For the like/dislike measure for each party, the respondents reported, on a seven-point scale, how much they liked or disliked each of the nine major Norwegian parties from (1) strongly dislike to (7) strongly like. We recoded these as either liking (5–7) or disliking (1–3) the party mentioned in the treatment. We coded the party as “neither” if the respondent neither liked nor disliked the party mentioned in the treatment.

Disentangling Recommender Attributes and News-Story Attributes

The question of how the causal treatment effect of recommender attributes affects those of the news-story attributes is inherently a counterfactual question; that is, the question concerns not only what is but also what could have been. Choice-based conjoint designs are well-suited to study whether the effects of the included attributes are conditional upon specific attributes and whether the result is conditional on which attributes and treatments are included (Dafoe, Zhang, and Caughey Citation2018). We follow the approach of Acharya, Blackwell, and Sen (Citation2018) for dealing with the causal effects of randomly including or excluding information. We test the effects of randomly including or excluding recommender attributes on the effects of the news-story attributes. That way, we can test whether the effects of the latter are contingent upon the attributes of the former.

As illustrated in Figure 1(e) (the between-subject randomization process), we investigated the moderation effects of the recommender attributes on selective exposure and selective sharing by splitting the sample into six groups: Respondents were randomly assigned to choosing among the recommender attributes only (Figure 1(a)), the news-story attributes only (Figure 1(b)), or both (Figure 1(a,b)). Furthermore, respondents were randomly assigned to either a dependent variable of exposure (from whom they would most likely select to read endorsed news stories; see Figure 1(c)) or a dependent variable of sharing (whose news stories they would most likely choose to recommend; see Figure 1(d)) in a social media setting. Because we wanted to account for the social cost of sharing information on SNSs compared to reading information, we operationalized these decisions as a commitment to read or share the next 10 stories. Thus, our outcome of interest is from which randomly generated profile (i.e., information about news stories, information about the recommender, or both) respondents would commit to either read or recommend information.

This approach has two key advantages that necessarily also create a limitation. First, the design allows us to investigate the treatment effects of a range of different attributes simultaneously, thus taking into account that there is a range of different pieces of information that can shape the likelihood of individuals sharing or reading a news story on SNSs. Second, as the design is split into six groups, in which some respondents randomly receive extensive information and others randomly receive limited information, we can explore counterfactual scenarios that again allow us to disentangle and compare the effects of recommender attributes with news-story attributes. While this is key to understanding, comparing, and distinguishing these causal treatment effects, it also creates a less realistic scenario for the respondents. Thus, there is a trade-off between the ability to compare and disentangle causal effects and the ability to strengthen ecological validity. That said, because our conjoint experiment includes a total of 10 attributes (in the group receiving all treatments), the decisions we measured also come closer to real-life decision situations than in traditional experiments in terms of available information as the respondents received relatively more information, thus taking into account that news-selection decisions are multidimensional.

Analysis

We measure from which profiles the respondents would and would not prefer to read or share news, and we code the responses as a binary variable: one if the profile was chosen and 0 if not. This choice is our outcome of interest. In the analysis of the results, we use the statistical approach developed by Hainmueller, Hopkins, and Yamamoto (Citation2014) and estimate the average marginal component effects (AMCE). In the present study, the AMCE shows the average difference in the probability of a profile being selected. Each component in the profiles represents attributes with different levels that are compared to a different attribute level within the same attribute. That way, we can assess if one attribute level is more or less important for the respondents’ selection decisions.

In order to measure moderation effects as well as differences between sharing and exposure decisions, we calculate the difference in the AMCE (contrasting the effect of one attribute with the effect of the same attribute under another condition). For instance, we regard recommender attributes as moderating the effects of news-story attributes if the effect of these attributes is statistically significantly higher or lower when respondents also received information about the recommender vs. information about the news story.

Differences between Reading and Sharing News on SNSs

We start by addressing whether, on SNSs, the likelihood of committing to sharing recommended news stories and that of committing to reading them differ. Figures 2 and 3 compare the effects of the likelihood of reading and sharing information when respondents are either only given information about the recommender (Figure 3) or only information about the shared news stories (Figure 2). For all figures presented in this article, dots indicate point estimates, bars illustrate 95% confidence intervals, and dots without bars are reference categories. All figures are presented in table format in the Appendix (Supplementary material).

Figure 2. Comparing effects (AMCE) of attributes (news-story attributes) on probability to read vs. probability to share.

Note: The dots represent the point estimates, and the bars show 95 percent confidence intervals.

Display full size

Figure 3. Comparing effects (AMCE) of attributes (recommender attributes) on probability to read vs. probability to share.

Note: The dots represent the point estimates, and the bars show 95 percent confidence intervals.

Display full size

Starting with the group of respondents who only received information about the shared news stories (Figure 2), we observe statistically significant treatment effects of message alignment for sharing but not for reading recommended news stories. The respondents were 14 percentage points more likely to share news stories if they mostly agreed with the news angle. Addressing the research question on differences between sharing and reading (RQ1), we observe that the likelihood of sharing like-minded news stories is 13 percentage points higher than that of reading like-minded news stories; this difference is statistically significant. In terms of party favorability, Figure 2 shows that respondents were 21 percentage points more likely to prefer a news story featuring a party they liked. This effect was similar (25 percentage points) for sharing decisions, and the difference between recommending and reading a news story was not statistically significantly different (RQ2). The only noteworthy and statistically significant difference between reading and sharing a news story based on party favorability attributes was that respondents were more likely to share (20 percentage points), but not to read (three percentage points) about a party they neither liked nor disliked. It is also important to note that respondents were less likely to both read and share a news story if the themes were gay rights or crime and punishment and more likely if the themes were immigration, European politics, or the economy. This indicates that while message alignment and party favorability are important attributes for reading and sharing news, the themes of the news stories are just as important.

Turning to the group that only received information about the recommender attributes, we observe that the effects of the recommender are strikingly similar for reading and sharing the news stories. For both decisions, tie strength, the recommender’s language skills, political knowledge, and political similarity in terms of party preference are crucial for committing to reading or recommending future news stories from that person. For instance, respondents are 25 percentage points more likely to read and 24 percentage points more likely to share news content from a politically knowledgeable recommender. Interestingly, we find no statistically significant main effects of the recommender’s popularity on social media, gender, or religious affiliation. In terms of the popularity of the recommender, this means that we find no indications of a bandwagon effect, in which respondents are more likely to share or read content when others on SNSs also tend to share the recommender’s recommended content.

When comparing the two decisions, we observe that for recommender attributes, the only statistically significantly different treatment effects on the two dependent variables are for tie-strength (RQ3). Close friends, compared to recommenders they have met through work or school, are more important for the decision to share (25 percentage points) than to read (13 percentage points) the news stories (a 12 percentage-point difference). In sum, we find few indications of stark differences between reading and sharing decisions, with attitude-alignment and tie-strength being two important exceptions.

Disentangling the Effects of Recommender Attributes

To address RQ7 and RQ8, we start by assessing whether the likelihood of committing to share news stories on SNSs featuring a party one favors vs. disfavors is contingent upon whether or not one also receives information about recommender attributes. Figure 4 shows the AMCE when respondents were only given information about the shared story, the AMCE when respondents received both information about the shared story and the recommender, and the difference in the AMCEs between the two groups. Starting with the group that only received information about the news stories, we observe a 25-percentage-point negative effect of disliking the party mentioned in the news stories. Similarly, we observe that respondents who only receive information about the recommender attributes are 20 percentage points less likely to share news articles from a recommender who supports a party the respondent dislikes. In the condition where respondents receive both of these treatments, however, the effect of the party in the news story disappears (reduced by 18 percentage points), while the effect of the recommender’s party preference is still present. This means that the party favorability effect provided by the recommender overrides the party favorability effect provided by the news story.

Figure 4. Moderation effects (conditional AMCEs) of attributes on profile selection (probability to share).

Note: The dots represent the point estimates, and the bars show 95 percent confidence intervals.

Display full size

Figure 4 also shows the treatment effects of agreeing with the shared news stories’ news angle. We observe that people are less likely to share information with which they mostly disagree rather than with which they equally agree and disagree or mostly agree. This effect is not changed by respondents also receiving information about the recommender.

Figure 5 shows the exact same treatment effects as displayed in Figure 4 but for the likelihood of reading the shared information. Here, we see an effect of party favorability of both the recommender and the story. However, in contrast to the news-sharing decision, the effect of party favorability of the story is not dependent on the effects of the party favorability of the recommender. This indicates that for the reading decision, the party favorability attributes of the story and the recommender yield additive effects as they are not changed by receiving information solely about the story, solely about the recommender, or about both. That said, the change in effect of party favorability follows the same patterns as for the sharing decisions; however, the difference (11 percentage points) is not statistically significant (as indicated by the difference plot to the right in Figure 5).

Figure 5. Moderation effects (conditional AMCEs) of attributes on profile selection (probability to read).

Note: The dots represent the point estimates, and the bars show 95 percent confidence intervals.

Display full size

Turning to the effect of reading news stories with which one mostly disagrees, we observe only a small but statistically significant effect of favoring like-minded news stories or a diet of equally like-minded and cross-cutting stories over exclusively cross-cutting stories, under conditions of receiving information about both the news story and the recommender. This effect is, however, not contingent upon receiving information about the recommender as the difference between the two conditions is not statistically significant (as indicated by the difference plot to the right in Figure 5).