Biology subjects

van Gerven, M. A. J.

Publications and source records attributed to van Gerven, M. A. J..

13 recordsLinked to original sources

bioRxiv2026

Brain alignment in deep neural networks emerges early and independently of object classification

Deep convolutional neural networks are leading models of biological vision, largely because of their strong brain alignment: their features predict neural responses better than earlier models. Yet they are believed to recognize objects differently, relying on texture where humans rely on shape and failing on perturbations humans handle effortlessly. What then does alignment reflect? Tracking three architectures densely during training, we find that alignment with human fMRI, EEG, and macaque electrophysiology is already largely present at initialization, when networks classify at chance, and reaches a plateau within one to five epochs; thereafter it changes only modestly while classification accuracy continues to climb to 75%. Network lesioning shows that a kernel's contribution to alignment is essentially uncorrelated with its contribution to classification throughout training. Brain-network alignment therefore appears to reflect the structure of the visual environment both systems encode, rather than a shared solution for classification.

neuroscience↗

bioRxiv2022

Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models

Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field. In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task. We show that 1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; 2) individual word decoding in reconstructed speech achieves 92-100% accuracy (chance level is 8%); 3) direct reconstruction from sensorimotor brain activity produces intelligible speech. These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

neuroscience↗

bioRxiv2021

Predictive coding is a consequence of energy efficiency in recurrent neural networks

Predictive coding represents a promising framework for understanding brain function. It postulates that the brain continuously inhibits predictable sensory input, ensuring a preferential processing of surprising elements. A central aspect of this view is its hierarchical connectivity, involving recurrent message passing between excitatory bottom-up signals and inhibitory top-down feedback. Here we use computational modelling to demonstrate that such architectural hard-wiring is not necessary. Rather, predictive coding is shown to emerge as a consequence of energy efficiency. When training recurrent neural networks to minimise their energy consumption while operating in predictive environments, the networks self-organise into prediction and error units with appropriate inhibitory and excitatory interconnections, and learn to inhibit predictable sensory input. Moving beyond the view of purely top-down driven predictions, we furthermore demonstrate, via virtual lesioning experiments, that networks perform predictions on two timescales: fast lateral predictions among sensory units, and slower prediction cycles that integrate evidence over time.

neuroscience↗

bioRxiv2020

End-to-end optimization of prosthetic vision

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a non-trivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments we show that such an approach is able to automatically find a task-specific stimulation protocol. The presented approach is highly modular and could be extended to dynamically optimize prosthetic vision for everyday tasks and requirements of the end-user.

bioengineering↗

bioRxiv2020

Visual Attention Through Uncertainty Minimization in Recurrent Generative Models

Allocating visual attention through saccadic eye movements is a key ability of intelligent agents. Attention is both influenced through bottom-up stimulus properties as well as top-down task demands. The interaction of these two attention mechanisms is not yet fully understood. A parsimonious reconciliation posits that both processes serve the minimization of predictive uncertainty. We propose a recurrent generative neural network model that predicts a visual scene based on foveated glimpses. The model shifts its attention in order to minimize the uncertainty in its predictions. We show that the proposed model produces naturalistic eye movements focusing on informative stimulus regions. Introducing additional tasks modulates the saccade patterns towards task-relevant stimulus regions. The models saccade characteristics correspond well with previous experimental data in humans, providing evidence that uncertainty minimization could be a fundamental mechanisms for the allocation of visual attention.

neuroscience↗

bioRxiv2019

Adaptive time scales in recurrent neural networks

Recurrent neural network models have become widely used in computational neuroscience to model the dynamics of neural populations as well as in machine learning applications to model data with temporal dependencies. The different variants of RNNs commonly used in these scientific fields can be derived as discrete time approximations of the instantaneous firing rate of a population of neurons. The time constants of the neuronal process are generally ignored in these approximations, while learning these time constants could possibly inform us about the time scales underlying temporal processes and enhance the expressive capacity of the network. To investigate the potential of adaptive time constants, we compare the standard Elman approximation to a more lenient one that still accounts for the time scales at which processes unfold. We show that such a model with adaptive time scales performs better on predicting temporal data, increasing the memory capacity of recurrent neural networks, and allows recovery of the time scales at which the underlying processes unfold.

neuroscience↗

bioRxiv2019

A large single-participant fMRI dataset for probing brain responses to naturalistic stimuli in space and time

Visual and auditory representations in the human brain have been studied with encoding, decoding and reconstruction models. Representations from convolutional neural networks have been used as explanatory models for these stimulus-induced hierarchical brain activations. However, none of the fMRI datasets currently available has adequate amounts of data for sufficiently sampling their representations. We recorded a densely sampled large fMRI dataset (TR=700 ms) in a single individual exposed to spatiotemporal visual and auditory naturalistic stimuli (30 episodes of BBCs Doctor Who). The data consists of 120.830 whole-brain volumes (approx. 23 h) of single-presentation data (full episodes, training set) and 1.178 volumes (11 min) of repeated narrative short episodes (test set, 22 repetitions), recorded with fixation over a period of six months. This rich dataset can be used widely to study the way the brain represents audiovisual input across its sensory hierarchies.

neuroscience↗

bioRxiv2019

Individual differences in the influence of mental imagery on conscious perception

Mental imagery and visual perception rely on similar neural mechanisms, but the function of this overlap remains unclear. One idea is that imagery can influence perception. Previous research has shown that imagining a stimulus prior to binocular presentation of rivalling stimuli increases the chance of perceiving the imagined stimulus. In this study we investigated how this effect interacts with bottom-up sensory input by comparing psychometric response curves for congruent and incongruent imagery in humans. A Bayesian hierarchical model was used, allowing us to simultaneously study group-level effects as well as effects for individual participants. We found strong effects of both imagery as well as its interaction with sensory evidence within individual participants. However, the direction of these effects were highly variable between individuals, leading to weak effects at the group level. This highlights the heterogeneity of conscious perception and emphasizes the need for individualized investigation of such complex cognitive processes.

neuroscience↗

bioRxiv2018

Classification of mouse ultrasonic vocalizations using deep learning

Vocalizations are a widespread means of communication in the animal kingdom. Mice use a large repertoire of ultrasonic vocalizations (USVs) in different social contexts, for instance courtship, territorial dispute, dominance and mother-pup interaction. Previous studies have pointed to differences in the USVs in different context, sexes, strains and individuals, however, in many cases the outcomes of the analyses remained inconclusive.\n\nWe here provide a more general approach to automatically classify USVs using deep neural networks (DNN). We classified the sex of the emitting mouse (C57Bl/6) based on the vocalizations spectrogram, reaching unprecedented performance (~84% correct) in comparison with other techniques (Support Vector Machines: 64%, Ridge regression: 52%). Vocalization characteristics of individual mice only contribute mildly, and sex-only classification reaches ~78%. The performance can only partially be explained by a set of classical shape features, with duration, volume and bandwidth being the most useful predictors. Splitting estimation into two DNNs, from spectrograms to features (57-82%) and features to sex (67%) does not reach the single-step performance.\n\nIn summary, the emitters sex can be successfully predicted from their spectrograms using DNNs, excelling over other classification techniques. In contrast to previous research, this suggests that male and female vocalizations differ in their spectrotemporal structure, recognizable even in single vocalizations.

bioinformatics↗

bioRxiv2017

Generative adversarial networks for reconstructing natural images from brain activity

We explore a method for reconstructing visual stimuli from brain activity. Using large databases of natural images we trained a deep convolutional generative adversarial network capable of generating gray scale photos, similar to stimUli presented during two functional magnetic resonance imaging experiments. Using a linear model we learned to predict the generative models latent space from measured brain activity. The objective was to create an image similar to the presented stimulus image through the previously trained generator. Using this approach we were able to reconstruct structural and some semantic features of a proportion of the natural images sets. A behavioral test showed that subjects were capable of identifying a reconstruction of the original stimuhis in 67.2% and 66.4% of the cases in a pairwise comparison for the two natural image datasets respectively. our approach does not require end-to-end training of a large generative model on limited neuroimaging data. Rapid advances in generative modeling promise further improvements in reconstruction performance.

neuroscience↗

bioRxiv2017

CNN-based Encoding and Decoding of Visual Object Recognition in Space and Time

Deep convolutional neural networks (CNNs) have been put forward as neurobiologically plausible models of the visual hierarchy. Using functional magnetic resonance imaging, CNN representations of visual stimuli have previously been shown to correspond to processing stages in the ventral and dorsal streams of the visual system. Whether this correspondence between models and brain signals also holds for activity acquired at high temporal resolution has been explored less exhaustively. Here, we addressed this question by combining CNN-based encoding models with magnetoencephalography (MEG). Human participants passively viewed 1000 images of objects while MEG signals were acquired. We modelled their high temporal resolution source-reconstructed cortical activity with CNNs, and observed a feedforward sweep across the visual hierarchy between 75-200 ms after stimulus onset. This spatiotemporal cascade was captured by the network layer representations, where the increasingly abstract stimulus representation in the hierarchical network model was reflected in different parts of the visual cortex, following the visual ventral stream. We further validated the accuracy of our encoding model by decoding stimulus identity in a left-out validation set of viewed objects, achieving state-of-the-art decoding accuracy.

neuroscience↗

bioRxiv2016

Slice-sampled Bayesian PRF mapping

Functional magnetic resonance imaging (FMRI) allows to non-invasively measure human brain activity at the millimeter scale. As such, it is widely used in computational neuroimaging studies that aim to build models to predict stimulus-induced neural responses in visual cortex. A popular method is population receptive field (PRF) mapping, which is able to characterize responses to a large range of stimuli. For each voxel, the PRF method estimates the best fitting receptive field properties (such as location and size in the visual field) using a coarse-to-fine approach which minimizes, but not eliminates, the risk of returning a local minimum. Here, we provide a Bayesian approach to the PRF method based on the slice sampler. Using this approach, we provide estimates of receptive field properties while at the same time being able to quantify their uncertainty. We test the performance of conventional and Bayesian approaches on simulated and empirical data.

neuroscience↗

bioRxiv2016

Modeling Cognitive Processes with Neural Reinforcement Learning

Artificial neural networks (ANNs) have seen renewed interest in the fields of computer science, artificial intelligence and neuroscience. Recent advances in improving the performance of ANNs open up an exciting new avenue for cognitive neuroscience research. Here, we propose that ANNs that learn to solve complex tasks based on reinforcement learning, can serve as a universal computational framework for analyzing the neural and behavioural correlates of cognitive processing. We demonstrate this idea on a challenging probabilistic categorization task, where neural network dynamics are linked to human behavioural and neural data as identical tasks are solved.

neuroscience↗