Brucklacher, M. | bioRxiv Science

bioRxiv2023

Learning to segment self-generated from externally caused optic flow through sensorimotor mismatch circuits

Efficient sensory detection requires the capacity to ignore task-irrelevant information, for example when optic flow patterns created by egomotion need to be disentangled from object perception. To investigate how this is achieved in the visual system, predictive coding with sensorimotor mismatch detection is an attractive starting point. Indeed, experimental evidence for sensorimotor mismatch signals in early visual areas exists, but it is not understood how they are integrated into cortical networks that perform input segmentation and categorization. Our model advances a biologically plausible solution by extending predictive coding models with the ability to distinguish self-generated from externally caused optic flow. We first show that a simple three neuron circuit produces experience-dependent sensorimotor mismatch responses, in agreement with calcium imaging data from mice. This microcircuit is then integrated into a neural network with two generative streams. The motor-to-visual stream consists of parallel microcircuits between motor and visual areas and learns to spatially predict optic flow resulting from self-motion. The second stream bidirectionally connects a motion-selective higher visual area (mHVA) to V1, assigning a crucial role to the abundant feedback connections: the maintenance of a generative model of externally caused optic flow. In the model, area mHVA learns to segment moving objects from the background, and facilitates object categorization. Based on shared neurocomputational principles across species, the model also maps onto primate vision. Our work extends the Hebbian predictive coding to sensorimotor settings, in which the agent actively moves - and learns to predict the consequences of its own movements. Significance statementThis research addresses a fundamental challenge in sensory perception: how the brain distinguishes between self-generated and externally caused visual motion. Using a computational model inspired by predictive coding and sensorimotor mismatch detection, the study proposes a biologically plausible solution. The model incorporates a neural microcircuit that generates sensorimotor mismatch responses, aligning with experimental data from mice. This microcircuit is integrated into a neural network with two streams: one predicting self-motion-induced optic flow and another maintaining a generative model for externally caused optic flow. The research advances our understanding of how the brain segments visual input into object and background, shedding light on the neural mechanisms underlying perception and categorization not only in rodents, but also in primates.

neuroscience↗

bioRxiv2022

Local minimization of prediction errors drives learning of invariant object representations in a generative network model of visual perception

The ventral visual processing hierarchy of the cortex needs to fulfill at least two key functions: Perceived objects must be mapped to high-level representations invariantly of the precise viewing conditions, and a generative model must be learned that allows, for instance, to fill in occluded information guided by visual experience. Here, we show how a multilayered predictive coding network can learn to recognize objects from the bottom up and to generate specific representations via a top-down pathway through a single learning rule: the local minimization of prediction errors. Trained on sequences of continuously transformed objects, neurons in the highest network area become tuned to object identity invariant of precise position, comparable to inferotemporal neurons in macaques. Drawing on this, the dynamic properties of invariant object representations reproduce experimentally observed hierarchies of timescales from low to high levels of the ventral processing stream. The predicted faster decorrelation of error-neuron activity compared to representation neurons is of relevance for the experimental search for neural correlates of prediction errors. Lastly, the generative capacity of the network is confirmed by reconstructing specific object images, robust to partial occlusion of the inputs. By learning invariance from temporal continuity within a generative model, despite little change in architecture and learning rule compared to static input- reconstructing Hebbian predictive coding networks, simply by shifting the training paradigm to dynamic inputs, the approach generalizes the predictive coding framework to dynamic inputs in a more biologically plausible way than self-supervised networks with non-local error-backpropagation. Author SummaryNeurons in the inferotemporal cortex of primates respond to images of complex objects independent of position, rotational angle, or size. While feedforward models of visual perception such as deep neural networks can explain this, they fail to account for the use of top-down information, for example when sensory evidence is scarce. Here, we address the question of how the neuronal networks in the brain learn both bottom-up and top-down processing without labels as they are used in the artificial supervised learning paradigm. Building on previous work that explains vision as a process of iteratively improving predictions, learning in the predictive coding network is driven by the local minimization of prediction errors. When trained on sequences of moving inputs, the network learns both invariant high-level representations comparable to those in the inferotemporal cortex of primates, and a generative model capable of reconstructing whole objects from partially occluded input images in agreement with experimental recordings from early visual areas. Advancing the search for experimental hallmarks of prediction errors, we find that error neurons in the higher areas of the network change their activity on a shorter timescale than representation neurons.

neuroscience↗