Risk of bias in Cochrane systematic reviews: assessments of risk related to attrition bias are highly inconsistent
BackgroundAn important part of the systematic review methodology is appraisal of the risk of bias in included studies. Cochrane systematic reviews (CSRs) are considered golden standard regarding systematic review methodology, but Cochranes instructions for assessing risk of attrition bias are vague, which may lead to inconsistencies in authors assessments. The aim of this study was to analyze consistency of judgments and support for judgments of attrition bias in CSRs of interventions published in the Cochrane Database of Systematic Reviews (CDSR).\n\nMethodsWe analyzed CSRs published from July 2015 to June 2016 in the CDSR. We extracted data on number of included trials, judgment of attrition risk of bias for each included trial (low, unclear or high) and accompanying support for the judgment (supporting explanation). We also assessed how many CSRs had different judgments for the same supporting explanations.\n\nResultsIn the main analysis we included 10292 judgments and supporting explanations for attrition bias from 729 CSRs. We categorized supporting explanations for those judgments into four categories and we found that most of the supporting explanations were unclear. Numerical indicators for percent of attrition, as well as statistics related to attrition were judged very differently. One third of CSR authors had more than one category of supporting explanation; some had up to four different categories. Inconsistencies were found even with the number of judgments, names of risk of bias domains and different judgments for the same supporting explanations in the same CSR.\n\nConclusionWe found very high inconsistency in methods of appraising risk of attrition bias in recent Cochrane reviews. Systematic review authors need clear guidance about different categories they should assess and judgments for those explanations. Clear instructions about appraising risk of attrition bias will improve reliability of the Cochranes risk of bias tool, help authors in making decisions about risk of bias and help in making reliable decisions in healthcare.