Roelandse, M. | bioRxiv Science

bioRxiv2024

Do organisms need an impact factor? Citations of key biological resources including model organisms reveal usage patterns and impact.

Research resources like transgenic animals and antibodies are the workhorses of biomedicine, enabling investigators to relatively easily study specific disease conditions. As key biological resources, transgenic animals and antibodies are often validated, maintained, and distributed from university based stock centers. As these centers heavily rely largely on grant funding, it is critical that they are cited by investigators so that usage can be tracked. However, unlike systems for tracking the impact of papers, the conventions and systems for tracking key resource usage and impact lag behind. Previous studies have shown that about 50% of the resources are not findable, making the studies they are supporting irreproducible, but also makes tracking resources difficult. The RRID project is filling this gap by working with journals and resource providers to improve citation practices and to track the usage of these key resources. Here, we reviewed 10 years of citation practices for five university based stock centers, characterizing each reference into two broad categories: findable (authors could use the RRID, stock number, or full name) and not findable (authors could use a nickname or a common name that is not unique to the resource). The data revealed that when stock centers asked their communities to cite resources by RRID, in addition to helping stock centers more easily track resource usage by increasing the number of RRID papers, authors shifted from citing resources predominantly by nickname ([~]50% of the time) to citing them by one of the findable categories ([~]85%) in a matter of several years. In the case of one stock center, the MMRRC, the improvement in findability is also associated with improvements in the adherence to NIH rigor criteria, as determined by a significant increase in the Rigor and Transparency Index for studies using MMRRC mice. From this data, it was not possible to determine whether outreach to authors or changes to stock center websites drove better citation practices, but findability of research resources and rigor adherence was improved.

bioinformatics↗

bioRxiv2020

Rigor and Transparency Index, a new metric of quality for assessing biological and medical science methods.

The reproducibility crisis in science is a multifaceted problem involving practices and incentives, both in the laboratory and in publication. Fortunately, some of the root causes are known and can be addressed by scientists and authors alike. After careful consideration of the available literature, the National Institutes of Health identified several key problems with the way that scientists conduct and report their research and introduced guidelines to improve the rigor and reproducibility of pre-clinical studies. Many journals have implemented policies addressing these same criteria. We currently have, however, no comprehensive data on how these guidelines are impacting the reporting of research. Using SciScore, an automated tool developed to review the methods sections of manuscripts for the presence of criteria associated with the NIH and other reporting guidelines, e.g., ARRIVE, RRIDs, we have analyzed ~1.6 million PubMed Central papers to determine the degree to which articles were addressing these criteria. The tool scores each paper on a ten point scale identifying sentences that are associated with compliance with criteria associated with increased rigor (5 pts) and those associated with key resource identification and authentication (5 pts). From these data, we have built the Rigor and Transparency Index, which is the average score for analyzed papers in a particular journal. Our analyses show that the average score over all journals has increased since 1997, but remains below five, indicating that less than half of the rigor and reproducibility criteria are routinely addressed by authors. To analyze the data further, we examined the prevalence of individual criteria across the literature, e.g., the reporting of a subject’s sex (21-37% of studies between 1997 and 2019), the inclusion of sample size calculations (2-10%), whether the study addressed blinding (3-9%), or the identifiability of key biological resources such as antibodies (11-43%), transgenic organisms (14-22%), and cell lines (33-39%). The greatest increase in prevalence for rigor criteria was seen in the use of randomization of subjects (10-30%), while software tool identifiability improved the most among key resource types (42-87%). We further analyzed individual journals over time that had implemented specific author guidelines covering rigor criteria, and found that in some journals, they had a big impact, whereas in others they did not. We speculate that unless they are enforced, author guidelines alone do little to improve the number of criteria addressed by authors. Our Rigor and Transparency Index did not correlate with the impact factors of journals.View Full Text

scientific communication and education↗