Vuksic, N. | bioRxiv Science

bioRxiv2024

Contrast-dependent response modulation in convolutional neural networks captures behavioral and neural signatures of visual adaptation

Human perception is robust under challenging conditions, for example when sensory inputs change over time. Temporal adaptation in the form of reduced responses to repeated external stimuli is ubiquitously observed in the brain, yet it remains unclear how repetition suppression aids recognition of novel inputs. To clarify this, we collected behavioural and electrocorticography (EEG) measurements while human participants categorized objects embedded in visual noise patterns after first viewing these patterns in isolation, inducing adaptation to the noise stimulus. We furthermore manipulated the availability of object information in the visual input by varying the contrast of the noise-embedded objects. Our results provide convergent behavioral, neural and computational evidence of a benefit of temporal adaptation on sensory representations. Adapting to a noise pattern resulted in overall faster object recognition and better recognition of objects as object contrast increased. These adaptation-induced behavioral improvements were accompanied by more pronounced contrast-dependent modulation of object-evoked EEG responses, and better decoding of object information from EEG activity. To identify potential neural computations mediating the benefits of temporal adaptation on object recognition, we equipped task-optimized deep convolutional neural networks (DCNNs) with different candidate mechanisms to adjust network activations over time. DCNNs with intrinsic adaptation mechanisms, such as additive suppression, best captured contrast-dependent human performance benefits, whilst also showing improved object decoding as a result of adaptation. Finally, adaptation effects in networks that use temporal divisive normalization, a biologically-plausible canonical neural computation, were most robust to spatial shifts, suggesting that temporal adaptation via divisive normalization aids stable representations of time-varying visual inputs. Overall, our results demonstrate how temporal adaptation improves sensory representations and identify candidate neural computations mediating these effects. Author summaryRobust perception is essential for the human brain to detect, process, and act upon new sensory inputs. Temporal adaptation is believed to play a key role in robust sensory processing by allowing neurons to continuously adjust their responses to previous inputs in order to optimize the processing of future inputs. Here, we show that temporal adaptation aids visual object recognition by improving neural representations of object contrast and object category. By emulating temporal adaptation in deep convolutional neural network models with different computational mechanisms, we identify candidate neural computations mediating benefits of temporal adaptation on sensory processing.

neuroscience↗

bioRxiv2024

Distinct representation of navigational action affordances in human behavior, brains and deep neural networks

To decide how to move around the world, we must determine which locomotive actions (e.g., walking, swimming, or climbing) are afforded by the immediate visual environment. The neural basis of our ability to recognize locomotive affordances is unknown. Here, we compare human behavioral annotations, functional magnetic resonance imaging (fMRI) measurements, and deep neural network (DNN) activations to both indoor and outdoor real-world images to demonstrate that human visual cortex represents locomotive action affordances in complex visual scenes. Hierarchical clustering of behavioral annotations of six possible locomotive actions show that humans group environments into distinct affordance clusters using at least three separate dimensions. Representational similarity analysis of multi-voxel fMRI responses in scene-selective visual cortex shows that perceived locomotive affordances are represented independently from other scene properties such as objects, surface materials, scene category or global properties, and independent of the task performed in the scanner. Visual feature activations from DNNs trained on object or scene classification as well as a range of other visual understanding tasks correlate comparatively lower with behavioral and neural representations of locomotive affordances than with object representations. Training DNNs directly on affordance labels or using affordance-centered language embeddings increases alignment with human behavior, but none of the tested models fully captures locomotive action affordance perception. These results uncover a new type of representation in the human brain that reflects locomotive action affordances. SignificanceTo navigate the world around us, we can use different actions, such as walking, swimming or climbing. How does our brain compute and represent such locomotive action affordances? Here, we show that activation patterns in high-level visual regions in the human brain represent information about affordances independent of other visual elements such as surface materials and objects, and do so in an automatic manner. We also demonstrate that commonly used models of visual processing in human brains, namely object- and scene- classification trained deep neural networks, do not strongly represent this information. Our results suggest that locomotive action affordance perception in scenes relies on specialized neural representations different from those used for other visual understanding tasks.

neuroscience↗