What is open peer review? A systematic review

1. Problems with peer review

Peer review is the formal quality assurance mechanism whereby scholarly manuscripts (e.g. journal articles, books, grant applications and conference papers) are made subject to the scrutiny of others, whose feedback and judgements are then used to improve works and make final decisions regarding selection (for publication, grant allocation or speaking time). This system is perhaps more recent than one might expect, with its main formal elements only in general use since the mid-twentieth century in scientific publishing (Spier, 2002). Researchers agree that peer review per se is necessary, but most find the current model sub-optimal. Ware’s 2008 survey, for example, found that an overwhelming majority (85%) agreed that “peer review greatly helps scientific communication” and that even more (around 90%) said their own last published paper had been improved by peer review. Yet almost two thirds (64%) declared that they were satisfied with the current system of peer review, and less than a third (32%) believed that this system was the best possible (Ware, 2008). A recent follow-up study by the same author reported a slight increase in the desire for improvements in peer review (Ware, 2016)

Widespread beliefs that the current model is sub-optimal can be attributed to the various ways in which traditional peer review has been subject to criticism. Peer review has been variously accused of:

In response to these criticisms, a wide variety of changes to peer review have been suggested (see the extensive overview in Walker & Rocha da Silva, 2015). Amongst these innovations, many have been labelled as “open peer review” at one time or another.

2. The contested meaning of open peer review

The diversity of the definitions provided for open peer review can be seen by examining just two examples. The first one is, to my knowledge, the first recorded use of the phrase “open peer review”:

  “[A]n open reviewing system would be preferable. It would be more equitable and more efficient. Knowing that they would have to defend their views before their peers should provide referees with the motivation to do a good job. Also, as a side benefit, referees would be recognized for the work they had done (at least for those papers that were published). Open peer review would also improve communication. Referees and authors could discuss difficult issues to find ways to improve a paper, rather than dismissing it. Frequently, the review itself provides useful information. Should not these contributions be shared? Interested readers should have access to the reviews of the published papers.” (Armstrong, 1982)

  “[O]pen review makes submissions OA [open access], before or after some prepublication review, and invites community comments. Some open-review journals will use those comments to decide whether to accept the article for formal publication, and others will already have accepted the article and use the community comments to complement or carry forward the quality evaluation started by the journal. ” (Suber, 2012)

Within just these two examples, there are already a multitude of factors at play, including the removal of anonymity, the publishing of review reports, interaction between participants, crowdsourcing of reviews, and making manuscripts public pre-review, amongst others. But each of these are distinct factors, presenting separate strategies for openness and targeting differing problems. For example, disclosure of identities aims usually at increasing accountability and minimizing bias, c.f. “referees should be more highly motivated to do a competent and fair review if they may have to defend their views to the authors and if they will be identified with the published papers” (Armstrong, 1982). Publication of reports, on the other hand, also tackles problems of incentive (reviewers can get credit for their work) and wastefulness (reports can be consulted by readers). Moreover, these factors need not necessarily be linked, which is to say that they can be employed separately: identities can be disclosed without reports being published, and reports published with reviewer names withheld, for example.

This diversity has led many authors to acknowledge the essential ambiguity of the term “open peer review” (Hames, 2014; Sandewall, 2012; Ware, 2011). The major attempt thus far to bring coherence to this confusing landscape of competing and overlapping definitions, is Emily Ford’s paper “Defining and Characterizing Open Peer Review: A Review of the Literature” (Ford, 2013). Ford examined thirty-five articles to produce a schema of eight “common characteristics” of OPR: signed review, disclosed review, editor-mediated review, transparent review, crowdsourced review, prepublication review, synchronous review, and post-publication review. Unfortunately, however, Ford’s paper fails to offer a definitive definition of OPR, since despite distinguishing eight “common characteristics” of OPR, Ford nevertheless tries to reduce it to merely one: open identities: “Despite the differing definitions and implementations of open peer review discussed in the literature, its general treatment suggests that the process incorporates disclosure of authors’ and reviewers’ identities at some point during an article’s review and publication” (p. 314). Summing up her argument elsewhere, she says: “my previous definition … broadly understands OPR as any scholarly review mechanism providing disclosure of author and referee identities to one another” (Ford, 2015). But the other elements of her schema do not reduce to this one factor. Many definitions do not include open identities at all. This hence means that although Ford claims to have identified several features of OPR, she in fact is asserting that there is only one defining factor (open identity), which leaves us where we started. Ford’s schema is also problematic elsewhere: it lists “editor-mediated review” and “pre-publication review” as distinguishing characteristics, despite these being common traits of traditional peer review; it includes questionable elements such as the purely “theoretical” “synchronous review”; and some of its characteristics do not seem to be “base elements”, but complexes of other traits – for example, the definition of “transparent review” incorporates other characteristics such as open identities (which Ford terms “signed review”) and open reports (“disclosed review”).

Open identities

Open identity peer review, also known as signed peer review (Ford, 2013; Nobarany & Booth, 2015) and “unblinded review” (Monsen & Horn, 2007), is review where authors and reviewers are aware of each other’s identities. Traditional peer review operates as either “single-blind”, where authors do not know reviewers’ identities, or “double-blind”, where both authors and reviewers remain anonymous. Double-blind reviewing is more common in the Arts, Humanities and Social Sciences than it is in STEM (science, technology, engineering and medicine) subjects, but in all areas single-blind review is by far the most common model (Walker & Rocha da Silva, 2015). A main reason for maintaining author anonymity is that it is assumed to tackle possible publication biases against authors with traditionally feminine names, from less prestigious institutions or non-English speaking regions (Budden et al., 2008; Ross et al., 2006). Reviewer anonymity, meanwhile, is presumed to protect reviewers from undue influence, allowing them to give candid feedback without fear of possible reprisals from aggrieved authors. Various studies have failed to show that such measures increase review quality, however (Fisher et al., 1994; Godlee et al., 1998; Justice et al., 1998; McNutt et al., 1990; van Rooyen et al., 1999). As Godlee and her colleagues have said, “Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors. Such measures are unlikely to improve the quality of peer review reports” (Godlee et al., 1998). Moreover, factors such as close disciplinary communities and internet search capabilities, mean that author anonymity is only partially effective, with reviewers shown to be able to identify authors in between 26 and 46 percent of cases (Fisher et al., 1994; Godlee et al., 1998).

Proponents of open identity peer review argue that it will enhance accountability, further enable credit for peer reviewers, and simply make the system fairer: “most importantly, it seems unjust that authors should be “judged” by reviewers hiding behind anonymity” (van Rooyen et al., 1999). Open identity peer review is argued, moreover, to potentially increase review quality, as it is theorised that reviewers will be more highly motivated and invest more care in their reviews if their names are attached to them. Opponents counter this by arguing that signing will lead to poorer reviews, as reviewers temper their true opinions to avoid causing offence. To date, studies have failed to show any great effect in either direction (McNutt et al., 1990; van Rooyen et al., 1999; van Rooyen et al., 2010). However, since these studies derive from only one disciplinary area (medicine), the results cannot taken as representative and hence further research is undoubtedly required.

Open reports

Open reports peer review is where review reports (either full reports or summaries) are published alongside the relevant article. The main benefits of this measure is in making currently invisible but potentially useful scholarly information available for re-use. There is increased transparency and accountability that comes with being able to examine normally behind-the-scenes discussions and processes of improvement and assessment, and a potential to further incentivize peer reviewers by making their peer review work a more visible part of their scholarly activities (thus enabling reputational credit).

Reviewing is hard work. Research Information Network reported in 2008 that a single peer review takes an average of four hours, at an estimated total annual global cost of around £1.9 billion (Research Information Network, 2008). Once an article is published, however, these reviews usually serve no further purpose than to reside in publisher’s long-term archives. Yet those reviews contain information that remains potentially relevant and useful in the here-and-now. Often, works are accepted despite the lingering reservations of reviewers. Published reports can enable readers to consider these criticisms themselves, and “have a chance to examine and appraise this process of ”creative disagreement" and form their own opinions” (Peters & Ceci, 1982). Making reviews public in this way also adds another layer of quality assurance, as the reviews are open to the scrutiny of the wider scientific community. Moreover, publishing reports also aims at raising the recognition and reward of the work of peer reviewers. Adding review activities to the reviewer’s professional record is common practice; author identification systems currently also add mechanisms to host such information (e.g. via ORCID) (Hanson et al., 2016). Finally, open reports give young researchers a guide (to tone, length, the formulation of criticisms) to help them as they begin to do peer review themselves.

The evidence-base against which to judge such arguments is not great enough to enable strong conclusions, however. Van Rooyen and her colleagues found that open reports correlate with higher refusal rates amongst potential reviewers, as well as an increase in time taken to write review but no concomitant effect on review quality (van Rooyen et al., 2010). Nicholson and Alperin’s small survey, however, found generally positive attitudes: “researchers … believe that open review would generally improve reviews, and that peer reviews should count for career advancement” (Nicholson & Alperin, 2016).

Open participation

Open participation peer review, also known as “crowdsourced peer review” (Ford, 2013; Ford, 2015), “community/public review” (Walker & Rocha da Silva, 2015) and “public peer review” (Bornmann et al., 2012), allows the wider community to contribute to the review process. Whereas in traditional peer review editors identify and invite specific parties (peers) to review, open participation processes invite interested members of the scholarly community to participate in the review process, either by contributing full, structured reviews or shorter comments. It may be that comments are open to anybody (anonymous or registered), or some credentials might first be required (e.g., Science Open requires an ORCID profile with at least five published articles). Open participation is often used as a complement to a parallel process of solicited peer review. It aims to resolve possible conflicts associated with editorial selection of reviewers (e.g. biases, closed-networks, elitism) and possibly improve the reliability of peer review by increasing the number of reviewers (Bornmann et al., 2012). Reviewers can come from the wider research community, as well as those traditionally under-represented in scientific assessment, including representatives from industry or members of special-interest groups, for example patients in the case of medical journals (Ware, 2011). This has the potential to open the pool of reviewers beyond those identified by editors to include all potentially interested reviewers (including those from outside academia), and hence increase the number of reviewers for each publication (though in practice this is unlikely). Evidence suggests this practice could help increase the accuracy of peer review. For example, Herron (2012) produced a mathematical model of the peer review process which showed that “the accuracy of public reader-reviewers can surpass that of a small group of expert reviewers if the group of public reviewers is of sufficient size”, although only if the numbers of reader-reviewers exceeded 50.

Criticisms of open participation routinely focus on questions about reviewers’ qualifications to comment and the incentives for doing so. As Stevan Harnad has said: “it is not clear whether the self-appointed commentators will be qualified specialists (or how that is to be ascertained). The expert population in any given speciality is a scarce resource, already overharvested by classical peer review, so one wonders who would have the time or inclination to add journeyman commentary services to this load on their own initiative” (Harnad, 2000). Moreover, difficulties in motivating self-selecting commentators to take part and deliver useful critique have been reported. Nature, for example, ran an experiment from June to December 2006 inviting submitting authors to take part in an experiment where open participation would be used as a complement to a parallel process of solicited peer reviews. Nature judged the trial to have been unsuccessful due to the small number of authors wishing to take part (just 5% of submitting authors), the small number of overall comments (almost half of articles received no comments) and the insubstantial nature of most of the comments that were received (Fitzpatrick, 2011). At the open access journal Atmospheric Chemistry and Physics (ACP), which publishes pre-review discussion papers for community comments, only about one in five papers is commented upon (Pöschl, 2012). Bornmann et al. (2012) conducted a comparative content analysis of the ACP’s community comments and formal referee reviews and concluded that the latter – tending to focus more on formal qualities, conclusions and potential impact – better supported the selection and improvement of manuscripts. This all suggests that although open participation might be a worthwhile complement to traditional, invited peer review, it is unlikely to be able to fully replace it.

Open interaction

Open interaction peer review allows and encourages direct reciprocal discussion between reviewers, and/or between author(s) and reviewers. In traditional peer review, reviewers and authors correspond only with editors. Reviewers have no contact with other reviewers, and authors usually have no opportunity to directly question or respond to reviewers’ comments. Allowing interaction amongst reviewers or between authors and reviewers, or between reviewers themselves, is another way to “open up” the review process, enabling editors and reviewers to work with authors to improve their manuscript. The motivation for doing so, according to (Armstrong, 1982), is to “improve communication. Referees and authors could discuss difficult issues to find ways to improve a paper, rather than dismissing it”.

Some journals enable pre-publication interaction between reviewers as standard (Hames, 2014). The EMBO Journal, for example, enables “cross-peer review,” where referees are “invited to comment on each other’s reports, before the editor makes a decision, ensuring a balanced review process” (EMBO Journal, 2016). At eLife, reviewers and editor engage in an “online consultation session” where they come to a mutual decision before the editor compiles a single peer review summary letter for the author to give them a single, non-contradictory roadmap for revisions (Schekman et al., 2013). The publisher Frontiers has gone a step further, including an interactive collaboration stage that “unites authors, reviewers and the Associate Editor – and if need be the Specialty Chief Editor – in a direct online dialogue, enabling quick iterations and facilitating consensus” (Frontiers, 2016).

Perhaps even more so than other areas studied here, evidence to judge the effectiveness of interactive review is scarce. Based on anecdotal evidence, Walker & Rocha da Silva (2015) advise that “[r]eports from participants are generally but not universally positive”. To the knowledge of the author, the only experimental study that has specifically examined interaction among reviewers or between reviewers and authors is that of Jeffrey Leek and his colleagues, who performed a laboratory study of open and closed peer review based on an online game and found that “improved cooperation does in fact lead to improved reviewing accuracy. These results suggest that in this era of increasing competition for publication and grants, cooperation is vital for accurate evaluation of scientific research” (Leek et al., 2011). Such results are encouraging, but hardly conclusive. Hence, there remains much scope for further research to determine the impact of cooperation on the efficacy and cost of the review process.

Open pre-review manuscripts

Open pre-review manuscripts are manuscripts that are immediately openly accessible (via the internet) in advance, or in synchrony with, any formal peer review procedures. Subject-specific “preprint servers” like arXiv.org and bioRxiv.org, institutional repositories, catch-all repositories like Zenodo or Figshare and some publisher-hosted repositories (like PeerJ Preprints) allow authors to short-cut the traditional publication process and make their manuscripts immediately available to everyone. This can be used as a complement to a more traditional publication process, with comments invited on preprints and then incorporated into redrafting as the manuscript goes through traditional peer review with a journal. Alternatively, services which overlay peer-review functionalities on repositories can produce functional publication platforms at reduced cost (Boldt, 2011; Perakakis et al., 2010). The mathematics journal Discrete Analysis, for example, is an overlay journal whose primary content is hosted on arXiv (Day, 2015). The recently released Open Peer Review Module for repositories, developed by Open Scholar in association with OpenAIRE, is an open source software plug-in which adds overlay peer review functionalities to repositories using the DSpace software (OpenAIRE, 2016). Another innovative model along these lines is that of ScienceOpen, which ingests articles metadata from preprint servers and contextualizes them by adding altmetrics and other relational information, before offering authors peer review.

In other cases, manuscripts are submitted to publishers in the usual way but made immediately available online (usually following some rapid preliminary review or “sanity check”) before the start of the peer review process. This approach was pioneered with the 1997 launch of the online journal Electronic Transactions in Artificial Intelligence (ETAI), where a two-stage review process was used. First, manuscripts were made available online for interactive community discussion, before later being subject to standard anonymous peer review. The journal stopped publishing in 2002 (Sandewall, 2012). Atmospheric Chemistry and Physics uses a similar system of multi-stage peer review, with manuscripts being made immediately available as “discussion papers” for community comments and peer review (Pöschl, 2012). Other prominent examples are F1000Research and the Semantic Web Journal.

The benefits to be gained from open pre-review manuscripts is that researchers can assert their priority in reporting findings – they needn’t wait for the sometimes seemingly endless peer review and publishing process, during which they might fear being scooped. Moreover, getting research out earlier increases its visibility, enables open participation in peer review (where commentary is open to all), and perhaps even, according to (Pöschl, 2012), increases the quality of initial manuscript submissions.

Open final-version commenting

Open final-version commenting is review or commenting on final “version of record” publications. If the purpose of peer review is to assist in the selection and improvement of manuscripts for publication, then it seems illogical to suggest that peer review can continue once the final version-of-record is made public. Nonetheless, in a literal sense, even the declared fixed version-of-record continues to undergo a process of improvement (occasionally) and selection (perpetually).

The internet has hugely expanded the range of effective action available for readers to offer their feedback on scholarly works. Where before only formal routes like the letters to the journal or commentary articles offered readers a voice, now a multitude of channels exist. Journals are increasingly offering their own commentary sections. Walker & Rocha da Silva (2015) found that of 53 publishing venues reviewed, 24 provided facilities to enable user-comments on published articles – although these were typically not heavily used. Researchers seem to see the worth of such functionalities, with almost half of respondents to a 2009 survey believing supplementing peer review with some form of post-publication commentary to be beneficial (Mulligan et al., 2013). But users can “publish” their thoughts anywhere on the Web – via academic social networks like Mendeley, ResearchGate and Academia.edu, via Twitter, or on their own blogs. The reputation of a piece of work is continuously evolving as long as it remains the subject of discussion.

Improvements based on feedback happen most obviously in the case of so-called ‘living’ publications, like the Living Reviews group of three disciplinary journals in the fields of relativity, solar physics and computational astrophysics, publishing invited review articles which allow authors to regularly update their articles to incorporate the latest developments in the field. Here, even where the published version is anticipated to be the final version, it remains open to future retraction or correction. Such changes are often fueled by social media, as in the 2010 case of #arseniclife, where social media critique over flaws in the methodology of a paper claiming to show a bacterium capable of growing on arsenic resulted in refutations being published in Science. The Retraction Watch blog is dedicated to publicizing such cases.

A major influence here has been the independent platform Pubpeer which proclaims itself a “post-publication peer review platform”. When its users swarmed to critique a Nature paper on STAP (Stimulus-Triggered Acquisition of Pluripotency) cells, PubPeer argued that its “post-publication peer review easily outperformed even the most careful reviewing in the best journal. The papers’ comment threads on PubPeer have attracted some 40000 viewers. It’s hardly surprising they caught issues that three overworked referees and a couple of editors did not. Science is now able to self-correct instantly. Post-publication peer review is here to stay” (PubPeer, 2014).

Open platforms

Open platforms peer review is review facilitated by a different organizational entity than the venue of publication. Recent years have seen the emergence of a group of dedicated platforms which aim to augment the traditional publishing ecosystem by de-coupling review functionalities from journals. Services like RUBRIQ and Peerage of Science offer “portable” or “independent” peer review. A similar service, Axios Review, operated from 2013 to 2017. Each platform invites authors to submit manuscripts directly to them, then organises review amongst their own community of reviewers and returns review reports. In the case of RUBRIQ and Peerage of Science, participating journals then have access to these scores and manuscripts and so can contact authors with a publishing offer or to suggest submission. Axios meanwhile, directly forwarded the manuscript, along with reviews and reviewer identities, to the author’s preferred target journal. The models vary in their details – RUBRIQ, for example, pays its reviewers, whereas Axios operated on a community model where reviewers earned discounts on having their own work reviewed – but all aim in their ways to reduce inefficiencies in the publication process, especially the problem of duplication of effort. Whereas in traditional peer review, a manuscript could undergo peer review at several journals, as it is submitted and rejected, then submitted elsewhere, such services need just one set of reviews which can be carried over to multiple journals until a manuscript finds a home (hence “portable” review).

Other decoupled platforms aim at solving different problems. Publons seeks to address the problem of incentive in peer review by turning peer review into measurable research outputs. Publons collects information about peer review from reviewers and publishers to produce reviewer profiles which detail verified peer review contributions that researchers can add to their CVs. Overlay journals like Discrete Mathematics, discussed above, are another example of open platforms. Peter Suber (quoted in Cassella & Calvi, 2010) defines the overlay journal as “An open-access journal that takes submissions from the preprints deposited at an archive (perhaps at the author’s initiative), and subjects them to peer review…. Because an overlay journal doesn’t have its own apparatus for disseminating accepted papers, but uses the pre-existing system of interoperable archives, it is a minimalist journal that only performs peer review.” Finally, there are the many venues through which readers can now comment on already-published works (see also “open final-version commenting” above), including blogs and social networking sites, as well as dedicated platforms such as PubPeer.