Lost in a Large EEG Multiverse? Comparing Sampling Approaches for Representative Pipeline Selection

Abstract

The multiplicity of defensible strategies for processing and analysing data has been implicated as a core contributor to the replicability crisis, creating uncertainty about the robustness of a result to variations in data processing choices. This issue is exacerbated where a large number of data processing pipelines are defensible, and where there is great heterogeneity in the pipelines applied in the literature, such as in processing and analysing electroencephalography (EEG) signals. In a multiverse analysis, equally defensible pipelines are computed and the robustness of the result to these variations is reported. However, a large number of defensible pipelines is sometimes infeasible to compute exhaustively, and researchers rely on sampling approaches. In these cases, pipelines are sampled from the full multiverse and the robustness is reported across these samples, assuming that they are representative for the entire multiverse. However, different sampling methods may yield different robustness results, introducing what we term multiverse sampling uncertainty. To illustrate, we computed a 528-pipeline multiverse analysis on EEG-recordings during an emotion classification task aiming to predict extraversion scores from the Late Positive Potential. We applied three sampling methods (random, stratified, and active learning) to sample 26 pipelines (5%), and evaluated the results in terms of the representativeness of the distribution of model fits to that of the full multiverse. Our results highlight variability in the representativeness of the distribution of model fits between samples. The active learning sample most closely represented the median model fit of the full multiverse. The need for representative pipeline sampling to mitigate bias in large multiverse analyses is discussed.