Stimulus overlap confounds generalization in EEG foundation-model benchmarks

Abstract

Neural decoding benchmarks commonly evaluate generalization across participants while reusing the same experimental stimuli in training and test sets. This design cannot distinguish transfer to new stimuli from recognition of familiar material, a distinction that is increasingly important for electroencephalography (EEG) foundation models, pretrained models designed to learn reusable neural representations. Here we introduce a training-matched audit that varies stimulus overlap while holding test trials, training participants and stimulus counts fixed. Across ten tasks from seven EEG emotion datasets, stimulus overlap increased differential-entropy decoding accuracy in every task and accounted for 77% of above-chance seen-stimulus performance on FACED-9. Across representations, overlap-dependent gains tracked stimulus identifiability; for CBraMod and LaBraM, advantages over input-matched handcrafted features were concentrated on familiar stimuli. Architecture-matched random controls nevertheless showed transferable benefits of pretraining on unseen stimuli. These results separate participant generalization from stimulus generalization and provide a framework for evaluating both in neural decoding benchmarks.