Biology subjects

Seijdel, N.

Publications and source records attributed to Seijdel, N..

7 recordsLinked to original sources

bioRxiv2026

Speech clarity shapes auditory attention and visual-signal coupling during multimodal sentence comprehension

Face-to-face communication requires listeners to attend, integrate, and weigh multiple communicative signals, including auditory speech, mouth movements, and co-speech gestures. The contribution of these signals may depend on the reliability of auditory input and the informativeness of the available signals. We utilized rapid invisible frequency tagging (RIFT) with EEG to examine how participants attend to and integrate these different signals in clear and adverse listening conditions. Participants watched videos of an actress producing clear or noise-vocoded sentences. Auditory speech was amplitude-modulated at 58Hz, while the luminance of the gesture and mouth regions was frequency-tagged at 63Hz and 65Hz. Degraded speech elicited stronger responses at the auditory tagged frequency, suggesting increased attentional gain to the auditory signal when listening was challenging. In contrast, clear speech elicited stronger responses at the gesture tagged frequency and a stronger 2Hz intermodulation response (65-63Hz), reflecting enhanced nonlinear coupling between mouth movements and gestures. Finally, in degraded speech, the informativeness of mouth movement, but not gesture, was associated with intermodulation strength, suggesting that the informativeness of mouth movements plays a greater role in multisensory interaction when listening is challenging. Our findings demonstrate that both signal reliability and informativeness shape multisensory integration during spoken language comprehension.

neuroscience↗

bioRxiv2023

Attention drives visual processing and audiovisual integration during multimodal communication

During communication in real-life settings, our brain often needs to integrate auditory and visual information, and at the same time actively focus on the relevant sources of information, while ignoring interference from irrelevant events. The interaction between integration and attention processes remains poorly understood. Here, we use rapid invisible frequency tagging (RIFT) and magnetoencephalography (MEG) to investigate how attention affects auditory and visual information processing and integration, during multimodal communication. We presented human participants (male and female) with videos of an actress uttering action verbs (auditory; tagged at 58 Hz) accompanied by two movie clips of hand gestures on both sides of fixation (attended stimulus tagged at 65 Hz; unattended stimulus tagged at 63 Hz). Integration difficulty was manipulated by a lower-order auditory factor (clear/degraded speech) and a higher-order visual semantic factor (matching/mismatching gesture). We observed an enhanced neural response to the attended visual information during degraded speech compared to clear speech. For the unattended information, the neural response to mismatching gestures was enhanced compared to matching gestures. Furthermore, signal power at the intermodulation frequencies of the frequency tags, indexing non-linear signal interactions, was enhanced in left frontotemporal and frontal regions. Focusing on LIFG, this enhancement was specific for the attended information, for those trials that benefitted from integration with a matching gesture. Higher power at this intermodulation frequency was related to faster reaction times. Together, our results suggest that attention modulates the strength and speed of audiovisual processing and interaction, depending on the congruence and quality of the sensory input.

neuroscience↗

bioRxiv2023

Human visual cortex and deep convolutional neural network care deeply about object background

Deep convolutional neural networks (DCNNs) are able to predict brain activity during object categorization tasks, but factors contributing to this predictive power are not fully understood. Our study aimed to investigate the factors contributing to the predictive power of DCNNs in object categorization tasks. We compared the activity of four DCNN architectures with electroencephalography (EEG) recordings obtained from 62 human subjects during an object categorization task. Previous physiological studies on object categorization have highlighted the importance of figure-ground segregation - the ability to distinguish objects from their backgrounds. Therefore, we set out to investigate if figure-ground segregation could explain DCNNs predictive power. Using a stimuli set consisting of identical target objects embedded in different backgrounds, we examined the influence of object background versus object category on both EEG and DCNN activity. Crucially, the recombination of naturalistic objects and experimentally-controlled backgrounds creates a sufficiently challenging and naturalistic task, while allowing us to retain experimental control. Our results showed that early EEG activity (<100ms) and early DCNN layers represent object background rather than object category. We also found that the predictive power of DCNNs on EEG activity is related to processing of object backgrounds, rather than categories. We provided evidence from both trained and untrained (i.e. random weights) DCNNs, showing figure-ground segregation to be a crucial step prior to the learning of object features. These findings suggest that both human visual cortex and DCNNs rely on the segregation of object backgrounds and target objects in order to perform object categorization. Altogether, our study provides new insights into the mechanisms underlying object categorization as we demonstrated that both human visual cortex and DCNNs care deeply about object background. Author summaryOur study aimed to investigate the factors contributing to the predictive power of deep convolutional neural networks (DCNNs) on EEG activity in object recognition tasks. We compared the activity of four DCNN architectures with human neural recordings during an object categorization task. We used a stimuli set consisting of identical target objects embedded in different phase-scrambled backgrounds. The distinction between object backgrounds and object categories allows us to investigate the influence of either factor for human subjects and DCNNs. Surprisingly, we found that both human visual processing and early DCNNs layers dedicate a large proportion of activity to processing object backgrounds instead of object category. Furthermore, this shared ability to make object backgrounds (and not just object category) invariant is largely the reason why DCNNs are predictive of brain dynamics in our experiment. We posit this shared ability to be an important solution for object categorization. Finally, we conclude that DCNNs, like humans, care deeply about object backgrounds.

neuroscience↗

bioRxiv2022

A critical test of deep convolutional neural networks ability to capture recurrent processing in the brain using visual masking

Recurrent processing is a crucial feature in human visual processing supporting perceptual grouping, figure-ground segmentation, and recognition under challenging conditions. There is a clear need to incorporate recurrent processing in deep convolutional neural networks (DCNNs) but the computations underlying recurrent processing remain unclear. In this paper, we tested a form of recurrence in deep residual networks (ResNets) to capture recurrent processing signals in the human brain. Though ResNets are feedforward networks, they approximate an excitatory additive form of recurrence. Essentially, this form of recurrence consists of repeating excitatory activations in response to a static stimulus. Here, we used ResNets of varying depths (reflecting varying levels of recurrent processing) to explain electroencephalography (EEG) activity within a visual masking paradigm. Sixty-two humans and fifty artificial agents (10 ResNet models of depths - 4, 6, 10, 18 and 34) completed an object categorization task. We show that deeper networks (ResNet-10, 18 and 34) explained more variance in brain activity compared to shallower networks (ResNet-4 and 6). Furthermore, all ResNets captured differences in brain activity between unmasked and masked trials, with differences starting at [~]98ms (from stimulus onset). These early differences indicated that EEG activity reflected pure feedforward signals only briefly (up to [~]98ms). After [~]98ms, deeper networks showed a significant increase in explained variance which peaks at [~]200ms, but only within unmasked trials, not masked trials. In summary, we provided clear evidence that excitatory additive recurrent processing in ResNets captures some of the recurrent processing in humans. Significance statementThe challenge of modeling recurrent processes is not trivial and the operationalization of recurrent processing is highly contested. In this paper, we tested the ability of deep residual networks (ResNets) to explain recurrent processes in the human brain. Though ResNets are feedforward networks, they have been shown to equate operations in recurrent neural networks. In this study, we show that deeper networks explained more variance in brain activity than shallower networks. However, all networks still performed far from the noise ceiling. Thus, we conclude that recurrent processing in ResNets captures a form of recurrent processing in humans though other types of recurrent processing (inhibition, multiplicative) that are not present in current regular deep neural networks (alexnet, cornet, resnet) are necessary for building better visual models.

neuroscience↗

bioRxiv2020

On the necessity of recurrent processing during object recognition: it depends on the need for scene segmentation

While feed-forward activity may suffice for recognizing objects in isolation, additional visual operations that aid object recognition might be needed for real-world scenes. One such additional operation is figure-ground segmentation; extracting the relevant features and locations of the target object while ignoring irrelevant features. In this study of 60 participants, we show objects on backgrounds of increasing complexity to investigate whether recurrent computations are increasingly important for segmenting objects from more complex backgrounds. Three lines of evidence show that recurrent processing is critical for recognition of objects embedded in complex scenes. First, behavioral results indicated a greater reduction in performance after masking objects presented on more complex backgrounds; with the degree of impairment increasing with increasing background complexity. Second, electroencephalography (EEG) measurements showed clear differences in the evoked response potentials (ERPs) between conditions around 200ms - a time point beyond feed-forward activity and object decoding based on the EEG signal indicated later decoding onsets for objects embedded in more complex backgrounds. Third, Deep Convolutional Neural Network performance confirmed this interpretation; feed-forward and less deep networks showed a higher degree of impairment in recognition for objects in complex backgrounds compared to recurrent and deeper networks. Together, these results support the notion that recurrent computations drive figure-ground segmentation of objects in complex scenes.

neuroscience↗

bioRxiv2019

Depth in convolutional neural networks solves scene segmentation

Feedforward deep convolutional neural networks (DCNNs) are, under specific conditions, matching and even surpassing human performance in object recognition in natural scenes. This performance suggests that the analysis of a loose collection of image features could support the recognition of natural object categories, without dedicated systems to solve specific visual subtasks. Research in humans however suggests that while feedforward activity may suffice for sparse scenes with isolated objects, additional visual operations ( routines) that aid the recognition process (e.g. segmentation or grouping) are needed for more complex scenes. Linking human visual processing to performance of DCNNs with increasing depth, we here explored if, how, and when object information is differentiated from the backgrounds they appear on. To this end, we controlled the information in both objects and backgrounds, as well as the relationship between them by adding noise, manipulating background congruence and systematically occluding parts of the image. Results indicate that with an increase in network depth, there is an increase in the distinction between object- and background information. For more shallow networks, results indicated a benefit of training on segmented objects. Overall, these results indicate that, de facto, scene segmentation can be performed by a network of sufficient depth. We conclude that the human brain could perform scene segmentation in the context of object identification without an explicit mechanism, by selecting or "binding" features that belong to the object and ignoring other features, in a manner similar to a very deep convolutional neural network.

neuroscience↗

bioRxiv2018

Scene complexity modulates degree of feedback activity during object recognition in natural scenes

Object recognition is thought to be mediated by rapid feed-forward activation of object-selective cortex, with limited contribution of feedback. However, disruption of visual evoked activity beyond feed-forward processing stages has been demonstrated to affect object recognition performance. Here, we unite these findings by reporting that the detection of target objects in natural scenes is selectively characterized by enhanced feedback when these objects are embedded in high complexity scenes. Human participants performed an animal target detection task on scenes with low, medium or high complexity as determined by a biologically plausible computational model of low-level contrast statistics. Three converging lines of evidence indicate that feedback was enhanced during categorization of scenes with high, but not low or medium complexity. First, functional magnetic resonance imaging (fMRI) activity in early visual cortex (V1) was selectively enhanced for target objects in scenes with high complexity. Second, event-related potentials (ERPs) evoked by high complexity scenes were selectively enhanced from 220 ms after stimulus-onset. Third, behavioral performance deteriorated for highly complex scenes when participants were pressed for time, but not when they could process the scenes fully and thereby benefit from the enhanced feedback. Formal modeling of the reaction time distributions revealed that object information accumulated more slowly for high complexity scenes (resulting in more errors especially for fast decisions), and directly related to the build-up of the feedback activity that was observed exclusively for high complexity scenes. Together, these results suggest that while feed-forward activity may suffice for simple scenes, the brain employs recurrent processing more adaptively in naturalistic settings, using minimal feedback for sparse, coherent scenes and increasing feedback for complex, fragmented scenes.\n\nAuthor summaryHow much neural processing is required to detect objects of interest in natural scenes? The astonishing speed of object recognition suggests that fast feed-forward buildup of perceptual activity is sufficient. However, this view is contradicted by findings that show that disruption of slower neural feedback leads to decreased detection performance. Our study unites these discrepancies by identifying scene complexity as a critical driver of neural feedback. We show how feedback is enhanced for complex, cluttered scenes compared to simple, well-organized scenes. Moreover, for complex scenes, more feedback is associated with better performances. These findings relate the flexibility of neural processes to perceptual decision-making by demonstrating that the brain dynamically directs neural resources based on the complexity of real-world visual inputs.

neuroscience↗