Excess significance bias in repetitive transcranial magnetic stimulation literature for neuropsychiatric disorders

Abstract

IntroductionRepetitive transcranial magnetic stimulation (rTMS) has been widely tested and promoted for use in multiple neuropsychiatric conditions, but as for many other medical devices, some gaps may exist in the literature and the evidence base for rTMS clinical efficacy remains under debate. We aimed to empirically test for an excess number of statistically significant results in the literature on rTMS therapeutic efficacy across a wide range of meta-analyses and to characterize the power of studies included in these meta-analyses.\n\nMethodsBased on power calculations, we computed the expected number of \"positive\" datasets for a medium effect-size (standardized mean difference, SMD=0.30) and compared it with the number of observed \"positive\" datasets. Sensitivity analyses considered small (SMD=0.20), modest (SMD=0.50), and large (SMD=0.80) effect sizes.\n\nResults14 meta-analyses with 228 datasets (110 for neurological disorders and 118 for psychiatric disorders) were assessed. For SMD=0.3, the number of observed \"positive\" studies (n=94) was larger than expected (n=35). We found evidence for an excess of significant findings overall (p<0.0001) and in 8/14 meta-analyses. Evidence for an excess of significant findings was also observed for SMD=0.5 for neurological disorders. 0 (0 %), 0 (0 %), 3 (1 %), and 53 (23 %) of the 228 datasets had power >0.80, respectively for SMDs of 0.30, 0.20, 0.50, and 0.80.\n\nConclusionMost studies in the rTMS literature are underpowered. This results in fragmentation and waste of research efforts. The somewhat high frequency of \"positive\" results seems spurious and may reflect bias.\n\nTrial Registration: PROSPERO 2017 CRD42017056694