Multimodal units fuse-then-accumulate evidence across channels

Abstract

Animals continuously detect information via multiple sensory channels, like vision and hearing, and integrate these signals to realise faster and more accurate decisions; a fundamental neural computation known as multisensory integration. A widespread view of this process is that multimodal neurons linearly fuse information across sensory channels. However, does linear fusion generalise beyond the classical tasks used to explore multisensory integration? Here, we develop novel multisensory tasks, which focus on the underlying statistical relationships between channels, and deploy models at three levels of abstraction: from probabilistic ideal observers to artificial and spiking neural networks. Using these models, we demonstrate that when information provided by different channels is not independent, linear fusion performs sub-optimally and even fails in extreme cases. This leads us to propose a simple nonlinear algorithm for multisensory integration which is compatible with our current knowledge of multimodal circuits, excels in naturalistic settings and is optimal for a wide class of multisensory tasks. Thus, our work emphasises the role of nonlinear fusion in multisensory integration, and provides testable hypotheses for the field to explore at multiple levels: from single neurons to behaviour. Key PointsO_LIWe introduce a novel set of tasks, based on comodulating the signals from multiple channels. C_LIO_LIWe show that linear fusion performs sub-optimally on these tasks and even fails in extreme cases. C_LIO_LIIn contrast, nonlinear fusion is optimal and excels in naturalistic settings, like predator-prey interactions. C_LIO_LIWe demonstrate that networks of spiking neurons approximate the behaviour of this nonlinear algorithm, when trained on these tasks. C_LIO_LIFinally, we explore how single neuron properties allow networks to approximate nonlinear fusion. C_LI