PubMed shuts down its comments feature, PubMed Commons

The U.S. National Institutes of Health is closing PubMed Commons, the feature that enabled readers to post comments on abstracts indexed in PubMed.

NIH announced it will be discontinuing the service — which allowed only signed comments from authors with papers indexed in PubMed, among other restrictions — after more than four years, due to a lack of interest.

According to the statement, the last day to post a comment will be February 15:

The service was first introduced as a pilot project in the fall of 2013 and was reviewed in 2015. Despite low levels of use at that time, NIH decided to extend the effort for another year or two in hopes that participation would increase. Unfortunately, usage has remained minimal, with comments submitted on only 6,000 of the 28 million articles indexed in PubMed.

While many worthwhile comments were made through the service during its 4 years of operation, NIH has decided that the low level of participation does not warrant continued investment in the project, particularly given the availability of other commenting venues.

Representatives of the NIH did not immediately respond to requests for comment. Comments will remain visible until March 2, after which readers will apparently be able to download them from the National Center for Biotechnology Information (NCBI) website.

According to a 2014 Nature article by NIH head Francis Collins and Lawrence Tabak, principal deputy director at the agency, the purpose of the feature was to encourage post-publication discussions about NIH-funded research, thereby improving its reproducibility.

So why didn’t the feature draw more interest?

The team at PubPeer, a site which allows anonymous comments on published literature, told Retraction Watch in a statement that they believed part of the problem with PubMed Commons was the restrictions placed on comments:

We believe that every obstacle to commenting reduces the volume of comments. Compared to PubPeer, PubMed Commons forbade anonymity. We have always felt that many users, particularly those with the most significant criticisms, prefer to comment anonymously in order to avoid any risk of reprisals. Irrespective of the true risk of such reprisals (and we have seen several legal threats in addition to the very public Sarkar suit), users do perceive a risk and alter their behavior in consequence. Anonymous commenting remains controversial, but we remain convinced that on balance it is beneficial.

PubPeer mirrors comments from PubMed Commons on its site, and said it plans to “do our best” to continue that, “although we are still exploring the technical aspects.” (Once the comments disappear from PubMed, users can reportedly see them directly on PubMed if they install PubPeer’s browser plugins.)

Nearly 70,000 publications have received comments on PubPeer since the site was established in 2012; the site currently displays more than 111,000 visible comments (54,000, excluding those stemming from PubMed Commons and Statcheck, an algorithm that checks for statistical errors in psychology studies).

From 2014 to 2016, more than 5,000 comments were posted to more than 4,000 publications, according to data presented at last year’s International Congress on Peer Review and Scientific Publication by three researchers based at the National Library of Medicine:

Most individual members of PubMed Commons have not commented, although a small number of members account for a considerable proportion of comments. Comments rarely include conflict of interest disclosures. Geographic distribution of commenters is not representative of authors in the biomedical literature, and women are underrepresented. Author replies are uncommon.

Dietram Scheufele of the University of Wisconsin-Madison, who has studied online discussions, told Retraction Watch he was partly “not surprised” that the feature did not take off. “The community is a bit split” on the value of post-publication review, he said, given that the discussions can quickly devolve:

As long as we haven’t figured out how to talk to each other in a civil and productive manner, this may be the best thing.

He added that simply seeing a comment isn’t enough to encourage readers to participate. Given that PubMed Commons included only signed comments, researchers had the ability to judge the person leaving the remark — and would probably be more likely to participate if they believed the discussion was being led by top people in the field (many of whom say they don’t have time to post comments).

It’s not just having comments – it’s having the comments from the best people who are going to push the agenda of my field forward.

When the feature launched in 2013, our co-founders explained why many sites — including Retraction Watch — allow anonymous comments:

By one measure, those opposed to anonymity are correct to say that “the quality of interchange would be higher if commenters were required to identify themselves.” Comments such as “This was a great contribution to the literature” are more likely. But the kind of respectful but critical questioning that helps correct the scientific record will be much less likely, because of realistic fears that senior scientists won’t like it very much when you criticize them. We’re once again struck that many journals and scientists seem to believe fervently in anonymous peer review to prevent this sort of revenge, but won’t allow anonymous comments because they make them uncomfortable.

Still, as the NIH statement notes, valuable discussions have emerged from PubMed Commons. For instance, in February 2017, a neuroscientist used the feature to alert readers there were duplication errors in his paper, then only days old; this April 2017 blog post from PubMed Commons lists other examples where authors have used the feature to alert readers to potential issues in their work. It’s even be used as a forum to debate credit for the invaluable lab tool, CRISPR. As the blog noted in December, 2015:

We believe the commenting function addresses a critical need, for PubMed and for the development of biomedical research. So a big “thank you” from us to everyone who has contributed their time and energy to supporting the Commons and commenting at PubMed.

Update Feb 2 2018, 22:29 UTC: We spoke with NLM deputy director Jerry Sheehan, who told us shutting down PubMed Commons:

… was the right decision to make for now, given the relatively low levels of use that we saw in the system.

He said although the site has plenty of traffic – millions of people go to PubMed each day – they clearly use it to search and find literature, not to comment on it.

It doesn’t fit into their workflows when they come to PubMed.

He noted that shutting down the feature doesn’t reflect at all on the reasons behind its creation:

Our commitment to transparency and reproducibility is unchanged, and we’re finding more effective mechanisms to achieve that.

Like Retraction Watch? You can make a tax-deductible contribution to support our growth, follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at retractionwatchteam@gmail.com.