Amplifying the impact of open access: Wikipedia and the diffusion of science

Correspondence Between Academic and Wikipedia Statuses

This article tests both the distillation and amplifier hypotheses by evaluating which references Wikipedia editors use and do not use. In particular we study how a journal's status within the scientific community (impact factor) and accessibility (open access policy) relates to its status within Wikipedia (percent of its articles referenced on Wikipedia). It is important to note that an observed correspondence may be driven by a variety of mechanisms besides accessibility. First, the status ordering of academic journals as measured with impact factors may have only a tenuous relationship with the importance and notability—considerations of special relevance to Wikipedia5—of the published research. Citations, and therefore impact factors, are in part a function of the research field (Seglen, 1997), and may be affected by factors as circumstantial as whether a paper's title contains a colon (Jamali & Nikzad, 2011; Seglen, 1997). Second, the academic status ordering results from the objectives of millions of scientists and institutions, and may be irrelevant to the unique objectives of Wikipedia. Wikipedia's key objective is to serve as an encyclopedia, not a medium through which scientists communicate original research6. Relative to the decentralization of the scientific literature, Wikipedia is governed by explicit, if flexible, policies and a hierarchical power structure (Butler, Joyce, & Pike, 2008; Shaw & Hill, 2014). Apart from a remark that review papers serve Wikipedia's objectives better than primary research articles, Wikipedia's referencing policies generally pass no judgment over which items within the scientific literature constitute “the best” evidence in support of a claim7. Wikipedia's objectives and explicit, centrally accessible, policies differ from the decentralized decisions that produce status orderings within the scientific literature and do not imply that the two status orderings should correspond. Indeed, if editors are not scientists themselves they need not even be aware that journal impact factors exist.8 On the other hand, despite the well-worn caveats, prestigious, high-impact journals may publish findings that are more important to both academics and Wikipedia's audience. In fact, a Wikipedia editor's expectation that the truly important research resides within high-impact journals may be enough to predispose them to want to reference such journals. Second, little is known about editors of science-related articles (West, Weber, & Castillo, 2012); they may be professional scientists with access to these high-impact journals, resulting in both the motivation and opportunity to reference them.

Wikipedia References and Academic Status

The first large-scale study of Wikipedia's scientific references was performed by Finn Arup Nielsen (Nielsen, 2007). Nielsen found that the number of Wikipedia references to the top 160 journals, extracted from the cite-journal citation templates, correlated modestly with the journal's Journal Citation Reports impact factor. This implication that Wikipedia preferentially cites high impact journals is delicate in part because the data used in the study included only a subset of journals with references that appear in Wikipedia, not journals that were and were not referenced. It is possible, albeit unlikely, that an even larger number of prestigious journals, made invisible by the methodology, are never referenced on Wikipedia at all, weakening the correlation to an unknown degree, or that the referenced journals are simply those that publish the most articles (see Nielsen 2007: Figure 1). Shuai et al. (2013) also found modest correlations when they investigated a possible correspondence between the academic rank of computer science papers, authors, and topics and their Wikipedia rank.

The altmetrics movement has also explored Wikipedia as a nonacademic venue on which academic literature makes an impact (ALM, Fenner, & Lin, ; “altmetrics,”; Priem, 2015). Evans and Krauthammer (2011) examined the use of Wikipedia as an alternative measure of the scholarly impact of biomedical research. The authors correlated scholarly metrics of biomedical articles, journals, and topics with Wikipedia citations and, in contrast to other studies, included in some of their analyses a random sample of journals never referenced on Wikipedia. The authors also recorded a journal's OA policy but, unfortunately, do not appear to have used this information in analyses.

Open Access and the Web

The rather voluminous literature on OA has focused primarily on effects on the academic literature.9 There is some debate on the size and direction of OA effects. Some evidence demonstrates that OA articles gain a citation advantage (Eysenbach, 2006a, 2006b; Gargouri et al., 2010; “The Open Access Citation Advantage Service”), whereas other evidence shows no such effect (Davis, 2011; Davis et al., 2008; Gaulé & Maystre, 2011; Moed, 2007). Regardless of the impacts on scientists in developed nations, increased accessibility through OA does yield benefits to scientists from developing nations (Davis & Walters, 2011; Evans & Reimer, 2009).

The promise of OA for disseminating scientific information to the world at large has gained much less attention (Davis & Walters, 2011; Trench, 2008; for an exception see Heilman & West, 2015). Yet, more and more of the world turns to the web for scientific information. For instance, as early as 1999 a full 20% of American adults sought medical and science information online (Miller, 2001). What's more, one who actively seeks such information within the academic literature will quickly discover that, despite the paywalls, many important and impactful research articles are made freely available by their authors or third parties (Björk, Laakso, Welling, & Paetau, 2014; Wren, 2005). This is to say nothing of the fact that science may also be disseminated through distillation of its findings into venues like Wikipedia or science-centric websites and blogs so that, here too, the impact of OA may be limited. Although full texts of the most impactful literature are, at least nominally, behind a paywall (Björk & Solomon, 2012), do Wikipedia's editors consult these texts? If they cite them in Wikipedia, have they consulted the full texts beyond a freely available abstract before referencing? If the academic literature is any guide, referenced material is sometimes consulted rather carelessly (Broadus, 1983; Rekdal, 2014). In short, the current understanding of the relationship between OA and the general public in the literature is limited at best (Davis & Walters, 2011).

Shortcomings and Our Contribution

In addition to the role of accessibility, a number of substantive and methodological shortcomings remain. First, it is unclear if professional scientists edit Wikipedia's science articles. As we will show, a preponderance of paywall references would suggest, albeit indirectly, this to be the case.10 The scant existing evidence indicates that science articles are edited by people with general expertise, relative to the more narrow experts of popular culture articles (West et al., 2012). Second, most previous studies have completely ignored the articles that are never referenced on Wikipedia, thus sampling on the dependent variable. The only notable exception, (Evans & Krauthammer, 2011), treated the unreferenced articles outside the main analytic framework. Although the framework treated (referenced) articles or journals as the unit of analysis, the unreferenced articles and journals were treated as a homogeneous group.

This study extends existing work in three chief ways. First, it models the role of accessibility (OA status) on referencing. Second, it covers all major research areas of science by observing rates at which Wikipedia references nearly 5,000 journals, accounting for nearly 20 million articles. Third, it treats unreferenced articles in the same analytic framework as those referenced. Yet the study is not without its own limitations, which will be outlined more fully later in the article. Chief among these are that article-level characteristics are operationalized by the characteristics of the publishing journal. For example, the accessibility of articles is operationalized by their journal's OA policy, when, in fact, free access to many paywall articles exists through sanctioned or unsanctioned file-sharing (Björk et al., 2014; Wren, 2005). Thus, any observed advantage of OA referencing may be biased downward, that is, an underestimate of the true effect (see the Conclusion for a discussion of measurement error).

Data Sample

Matching Scopus Journals to Wikipedia References

We checked each of the referenced journal names on Wikipedia against a list of Scopus-indexed journal names and common Institute for Scientific Information (ISI; now Thomson Reuters) journal name abbreviations. Of the 311,947 cite-journal tags, 203,536 could be linked to journals indexed by Scopus. Many of these references were nonunique, whereas our outcome of interest is whether articles from a journal are referenced on Wikipedia at all, not how many times. Therefore, to ensure that the counts for each journal included only unique articles, we distinguished articles by their DOIs and, if an article's DOI was not available, we used the article's title. Scopus's coverage of the output of various journals varies widely; our counts included only those articles published within the years of Scopus coverage.

In the end we matched 32,361 unique articles (and 55,262 total references) to our subset of Scopus journals (top 250 in each research field). Of the top Scopus journals, 2,005 are never referenced on the English Wikipedia. In most cases observed “journal names” that did not match to journals in Scopus were not academic journals but popular newspapers and magazines. Table A1 in the Appendix displays the 20 most frequently referenced sources that we were unable to link to Scopus. The top three nonscience sources are Billboard, National Park Service, and Royal Gazette. However, efforts to match Wikipedia references to Scopus were imperfect, and the list also includes a handful of academic journals, including The Lancet.

Journal Versus Article-Level Unit of Analysis

We chose to take journals instead of individual articles as our unit of analysis for several reasons. First and most important, accessibility of articles, the focal point of this study, was measured at the journal level by whether the journal is or is not OA. Switching the unit of analysis to individual articles would have simply assigned the same value of accessibility to all articles from a particular journal. Second, although article-level citations are an attractive, finely grained metric, a journal's impact factor is also designed to capture citation impact, albeit more coarsely. The general topic of any given article is also well captured by the host journal's Scopus -assigned topic(s). Lastly, the matching of Wikipedia journal title strings to Scopus required some manual matching and these efforts were more practical at the level of thousands of journals instead of hundreds of thousands of articles.

Percent_Cited and Other Variables

We present some of our results in terms of percent_cited—the percent of a journal's articles that are referenced on Wikipedia. An equivalent interpretation of this journal-level metric is the probability that a given article from a journal is referenced on Wikipedia. Figure 2 illustrates the distribution (kde) of percent_cited.

As Figure 1 demonstrates, the vast majority of journals that scientists use are referenced on the English Wikipedia very little: on average 0.19% of a journal's articles are referenced.15

As mentioned, the academic status of journals was measured using (SCImago) impact factors. To limit the influence of the few journals with uncommonly high impact factors the impact factor variable was (natural) log-transformed when used in the models. Figure 2 displays the distribution of impact factor and log-impact factor; to aid visualization only journals with impact factor <=15 are shown.

Table 2 presents the summary statistics for key variables: percent_cited, impact factor, ln (impact factor), and open access. Additionally, analyses use dummies for the 26 subject categories, for example, psychology 0 or 1).

Lastly, Figure 3 displays a scatter plot of the key dependent variable, percent_cited, versus impact factor and open access.

The scatter plots appear to show a modest relationship between a journal's impact factor and percent_cited, the percent of its articles referenced on Wikipedia, especially when considering journal size (dot size). The next section analyzes these relationships statistically.

Limitations and Directions for Future Research

Our findings are not without limitations. First and foremost, it bears emphasis that this study did not investigate the nature of Wikipedia's sources as a whole (see Ford, Sen, Musicant, & Miller, 2013 for an excellent examination of sources). Only a fraction of Wikipedia's references use the scientific literature, and this is the subset on which we focused. Consequently the present study cannot address the concern expressed by others, for example, (Luyt & Tan, 2010), that sources outside the scientific literature are used too heavily in scientific articles. Second, the study was cross-sectional in nature; it is conceivable that OA articles differ from closed access, paywall articles in their relevance to Wikipedia. Future work can test the potential confounding factor of unmeasured relevance by observing reference rates for articles which have been experimentally assigned to open and closed-access statuses, as has been done by some psychology journals (Davis et al., 2008).

Third, the study measured accessibility of articles by the OA policy of the publishing journals. However, many articles in paywall journals are made freely available by their authors or third parties (Björk et al., 2014; Wren, 2005). The resulting error in the measurement of accessibility may bias the observed advantage of OA in either direction: if OA articles from paywall journals, erroneously coded as closed access, are referenced at higher rates than the journals' truly closed access articles (Gargouri et al., 2010; Harnad & Brody, 2004), the true advantage of OA will be even higher than we observed. In the (unlikely) case that OA articles in paywall journals are referenced less than closed access articles, the observed open access advantage will be an overestimate. The academic status of articles is also operationalized by a journal characteristic—its impact factor. In fact, many articles out- or under-perform their journal's impact factor. Although this measurement error likely adds noise to the data, it probably does so without biasing the estimated effect of impact factor on referencing in one direction or another.

The Impact of Open Access Science

The chief finding of this study bears emphasis. We believe the existing discussion of open access has focused too narrowly on the academic literature. Early results showing that open access improves scientific outcomes such as citations have been tempered by newer experimental evidence showing small to null causal effects, and a lively debate has ensued. Our research shifts focus to diffusion, showing that open access policies have a tremendous impact on the diffusion of science to the broader general public through an intermediary like Wikipedia. This effect, previously a matter primarily of speculation, has empirical support. As millions of people search for scientific information on Wikipedia, the science they find distilled and referenced in its articles consists of a disproportionate quantity of open access science.