Brain2VLM: Hierarchical Alignment Between Cortical Representations and Vision-Language Latent Spaces
This work introduces Brain2VLM, a framework for analyzing how cortical representations align with latent spaces of pretrained diffusion-based vision-language models for brain-to-image reconstruction. While recent approaches achieve strong performance by mapping functional Magnetic Resonance Imaging (fMRI) signals to model latents, the structure of this mapping remains poorly understood. We hypothesize that brain-to-latent alignment is hierarchical, with early visual cortex exhibiting approximately linear correspondence to structural diffusion latents, and higher-order visual areas requiring nonlinear mappings to align with semantic embedding spaces. To test this, we decode diffusion latents and CLIP embeddings from fMRI signals across four subjects from the Natural Scenes Dataset (NSD) using both linear ridge regression and a nonlinear residual MLP. Our results reveal that nonlinear decoding provides only marginal improvements for diffusion latents ({Delta} {approx} 0.05 - 0.06 in correlation), but yields substantial gains for semantic embeddings ({Delta} {approx} 0.47), significantly improving distributional alignment (MMD: 0.042 vs 0.358). However, increased decoder expressivity can introduce shifts in latent distributions, highlighting a trade-off between prediction accuracy and generative compatibility. Despite using a simple reconstruction pipeline, Brain2VLM consistently improves semantic reconstruction quality across all four subjects while maintaining competitive pixel-level fidelity (PixCorr 0.36, CLIP 85%), suggesting that improvements in brain-to-latent alignment play an important role in reconstruction quality alongside generative modeling. These findings are consistent with a hierarchical pattern of alignment between cortical representations and model latent spaces, highlighting the brain-to-latent interface as an important component of brain decoding systems. Our code can be found at https://github.com/adarsh-crafts/Brain2VLM