Is biomedical research self-correcting? Modelling insights on the persistence of spurious science

Abstract

The reality that volumes of published research are not reproducible has been increasingly recognised in recent years, notably in biomedical science. In many fields, spurious results are common, reducing trustworthiness of reported results. While this increases research waste, a common response is that science is ultimately self-correcting, and trustworthy science will eventually triumph. While this is likely true from a philosophy of science perspective, it does not yield information on how much effort is required to nullify suspect findings, nor factors that shape how quickly science may be correcting in the publish-or-perish environment scientists operate. There is also a paucity of information on how perverse incentives of the publishing ecosystem, which reward novel positive findings over null results, shaping the ability of published science to self-correct. Precisely what factors shape self-correction of science remain obscure, limiting our ability to mitigate harms. This modelling study illuminates these questions, introducing a simple model to capture dynamics of the publication ecosystem, exploring factors influencing research waste, trustworthiness, corrective effort, and time to correction. Results from this work indicate that research waste and corrective effort are highly dependent on field-specific false positive rates and the time delay before corrective results to spurious findings are propagated. The model also suggests conditions under which biomedical science is self-correcting, and those under which publication of correctives alone cannot stem the propagation of untrustworthy results. Finally, this work models a variety of potential mitigation strategies, including researcher and publication driven interventions. Significance statementIn biomedical science, there is increasing recognition that many results fail to replicate, impeding both scientific advances and trust in science. While science is self-correcting over long time-scales, there has been little work done on the factors that shape time to correction, the scale of corrective efforts, and the research waste generated in these endeavours. Similarly, there has been little work done on quantifying factors that might reduce negative impacts of spurious science. This work takes a modeling approach to illuminate these questions, uncovering new strategies for mitigating the impact of untrustworthy research.