FUSE: FUsing EEG-MEG in a Shared Embedding via self-supervised learning for BCI

Abstract

Combining complementary neurophysiological modalities offers a promising strategy for improving motor imagery (MI) brain-computer interfaces (BCIs), but learning shared representations across modalities remains largely unexplored. Here, we propose a two-phase deep learning framework for multimodal EEG-MEG decoding that explicitly decouples representation learning from downstream classification. In the first phase, a convolutional encoder-decoder learns a shared latent representation by predicting the power spectral density (PSD) of EEG and MEG signals directly from time-domain activity, rather than using the conventional objective of reconstructing the input signal. In the second phase, the encoder is frozen and its learned representations are reused, without further adaptation, to perform the classification of the downstream MI-BCI task. The framework was evaluated on simultaneous EEG and MEG recordings from 20 participants. The learned representations consistently outperformed conventional handcrafted spectral features, increasing median classification accuracy from 0.734 to 0.794. The multimodal framework also improved performance over MEG alone (median accuracy from 0.680 to 0.794) and yielded a modest increase over EEG alone (median accuracy from 0.765 to 0.794), providing a more robust decoding strategy than single-modality approaches. Furthermore, the learned latent representations were transferable across participants, with more than half of cross-subject models performing within 0.01 accuracy of their subject-specific counterparts. These findings demonstrate that task-agnostic representation learning can capture physiologically meaningful multimodal neural representations that remain transferable across individuals, offering a promising foundation for more robust and reusable BCI pipelines.