The All Window-Size Search method for improved statistical power in multiple comparisons correction

Abstract

Correcting for multiple comparisons is a fundamental challenge throughout the biological sciences, particularly for data sampled over ordered continua such as time, space, or frequency. Existing approaches, including cluster-based permutation tests and threshold-free cluster enhancement (TFCE), leverage spatial or temporal contiguity but remain dependent on predefined statistical frameworks or thresholding procedures. Here we introduce the All Window-Size Search (AWSS) method, a permutation-based procedure that formally controls the family-wise error rate while adaptively searching across all contiguous window sizes and locations. For each permutation, test statistics are summed across every possible window, generating null distributions of maximal statistics at every window size. A second stage estimates the null distribution of the most significant uncorrected p-value that would arise from searching across all window sizes, allowing final p-values to be corrected for the adaptive search process itself. This procedure statistically formalizes the implicit multiscale search that investigators naturally perform when visually inspecting ordered data. Simulations with known ground-truth effects demonstrate that AWSS can provide substantially greater statistical power than conventional cluster-based permutation methods for broad, low-amplitude effects while maintaining appropriate family-wise error control. Because the framework is independent of any particular statistical test, it is readily applicable to diverse forms of one-dimensional ordered data. Here we test this application with simulations as well as using real human sEEG neural recording data. Future extensions will generalize the method to multidimensional spatial and spatiotemporal datasets, including neuroimaging and other high-dimensional biological data.