Biology subjects

Formisano, E.

Publications and source records attributed to Formisano, E..

8 recordsLinked to original sources

bioRxiv2026

A behaviourally normed database of 1,377 natural sounds for auditory cognition and neuroscience

Natural-sound research requires stimulus sets that combine acoustic standardization with detailed behavioural characterization. We present 1,377 two-second sounds representing 240 expert-defined source--action classes. We call this database "MaMa Sounds", as it resulted from the collaborative effort of two academic teams in Maastricht and Marseille. The sounds were manually curated, segmented, sampled at 16 kHz, and labelled with a noun identifying the source and a verb identifying the action. We release deidentified trial-level identification and familiarity data together with multiple per-sound norms (e.g., identification accuracy, confidence and agreement; familiarity), along with overall norms derived with principal component analysis. Noun, verb, and joint noun--verb norms are provided as direct means and medians with the number of contributing observations. This battery preserves process-specific information, while two principal-component scores provide compact overall behavioural-identifiability measures derived from response ease, semantic correspondence, agreement, and familiarity. The repository also contains deterministic response-cleaning code, participant and reference Word2Vec representations, and code reproducing the public sound-level tables. The resource supports stimulus selection, matching, and continuous modelling in auditory cognition and neuroscience.

neuroscience↗

bioRxiv2026

Multiscale Temporal Processing Supports Sound Recognition under Causal Constraints

The auditory system operates under a fundamental computational constraint: at any moment, it has access only to past and present acoustic information. At the same time, it processes sounds across multiple temporal scales, although the computational advantage of this organization remains unclear. Despite both being defining characteristics of biological auditory processing, causal processing and multiscale temporal processing are rarely considered together in computational models. Here, we introduce the Multiscale Convolutional Recurrent Neural Network (MSCRNN) architecture, a brain-inspired model designed to test whether processing sounds across multiple temporal contexts improves recognition under causal constraints. Multiscale processing substantially improved recognition under causal constraints, enabling performance comparable to non-causal architectures after an initial evidence-accumulation period while providing only limited benefit when future acoustic information was available. Analyses of the networks internal representations revealed a progressive transformation from stream-specific acoustic representations to increasingly integrated category-level representations across recurrent processing stages. Together, these findings provide a computational rationale for why biological auditory systems may benefit from multiscale temporal processing under causal constraints and establish the MSCRNN as an interpretable framework for generating and testing hypotheses about the temporal dynamics of auditory processing using electrophysiological and neuroimaging data.

neuroscience↗

bioRxiv2025

Deep Sound Synthesis Reveals Novel Category-Defining Sound Features in the Human Auditory Cortex

The human auditory system extracts meaning from the environment by transforming acoustic input signals into semantic categories. Specific acoustic features give rise to distinct categorical percepts, such as speech or music, and to spatially distinct preferential responses in the auditory cortex. These responses contain category-relevant information, yet their representational level and role within the acoustic-to-semantic transformation process remain unclear. We combined neuroimaging, a deep neural network, a brain-based sound synthesis, and psychophysics to identify the sound features that are internally represented in the speech- and music-selective human auditory cortex and test their functional role in sound categorization. We found that the synthetized sounds exhibit unnatural features distinct from those normally associated with speech and music, yet they elicit categorical cortical and behavioral responses resembling those of natural speech and music. Our findings provide new insights into the fundamental sound features underlying speech and music categorization in the human auditory cortex.

neuroscience↗

bioRxiv2024

Bridging Auditory Perception and Natural Language Processing with Semantically informed Deep Neural Networks

Sound recognition is effortless for humans but poses a significant chal-lenge for artificial hearing systems. Deep neural networks (DNNs), especially convolutional neural networks (CNNs), have recently sur-passed traditional machine learning in sound classification. However, current DNNs map sounds to labels using binary categorical variables, neglecting the semantic relations between labels. Cognitive neuroscience research suggests that human listeners exploit such semantic informa-tion besides acoustic cues. Hence, our hypothesis is that incorporating semantic information improves DNNs sound recognition performance, emulating human behavior. In our approach, sound recognition is framed as a regression problem, with CNNs trained to map spec-trograms to continuous semantic representations from NLP models (Word2Vec, BERT, and CLAP text encoder). Two DNN types were trained: semDNN with continuous embeddings and catDNN with cat-egorical labels, both with a dataset extracted from a collection of 388,211 sounds enriched with semantic descriptions. Evaluations across four external datasets, confirmed the superiority of semantic labeling from semDNN compared to catDNN, preserving higher-level relations. Importantly, an analysis of human similarity ratings for natural sounds, showed that semDNN approximated human listener behavior better than catDNN, other DNNs, and NLP models. Our work contributes to understanding the role of semantics in sound recognition, bridging the gap between artificial systems and human auditory perception.

neuroscience↗

bioRxiv2021

Listening to speech in noisy scenes: Antithetical contribution of primary and non-primary auditory cortex

Invasive and non-invasive electrophysiological measurements during "cocktail-party"-like listening indicate that neural activity in the human auditory cortex (AC) "tracks" the envelope of relevant speech. Due to the measurements limited coverage and/or spatial resolution, however, the distinct contribution of primary and non-primary auditory areas remains unclear. Using 7-Tesla fMRI, here we measured brain responses of participants attending to one speaker, without and with another concurrent speaker. Using voxel-wise modeling, we observed significant speech envelope tracking in bilateral Heschls gyrus (HG) and right middle superior temporal sulcus (mSTS), despite the sluggish fMRI responses and slow temporal sampling. Neural activity was either positively (HG) or negatively (mSTS) correlated to the speech envelope. Further analyses comparing the similarity between spatial response patterns in the concurrent speakers and single speaker conditions indicated that whereas tracking in HG reflected both relevant and (to a lesser extent) non-relevant speech, right mSTS selectively represented the relevant speech signal. Additionally, in right mSTS, the similarity strength correlated with the participants comprehension of relevant speech. These results indicate that primary and non-primary AC antithetically process ongoing speech suggesting a push-pull of acoustic and linguistic information.

neuroscience↗

bioRxiv2021

Training neural networks to recognize speech increased their correspondence to the human auditory pathway but did not yield a shared hierarchy of acoustic features

The correspondence between the activity of artificial neurons in convolutional neural networks (CNNs) trained to recognize objects in images and neural activity collected throughout the primate visual system has been well documented. Shallower layers of CNNs are typically more similar to early visual areas and deeper layers tend to be more similar to later visual areas, providing evidence for a shared representational hierarchy. This phenomenon has not been thoroughly studied in the auditory domain. Here, we compared the representations of CNNs trained to recognize speech (triphone recognition) to 7-Tesla fMRI activity collected throughout the human auditory pathway, including subcortical and cortical regions, while participants listened to speech. We found no evidence for a shared representational hierarchy of acoustic speech features. Instead, all auditory regions of interest were most similar to a single layer of the CNNs: the first fully-connected layer. This layer sits at the boundary between the relatively task-general intermediate layers and the highly task-specific final layers. This suggests that alternative architectural designs and/or training objectives may be needed to achieve fine-grained layer-wise correspondence with the human auditory pathway. HighlightsO_LITrained CNNs more similar to auditory fMRI activity than untrained C_LIO_LINo evidence of a shared representational hierarchy for acoustic features C_LIO_LIAll ROIs were most similar to the first fully-connected layer C_LIO_LICNN performance on speech recognition task positively associated with fmri similarity C_LI

neuroscience↗

bioRxiv2019

Dynamic time-locking mechanism in the cortical representation of spoken words

Human speech has a unique capacity to carry and communicate rich meanings. However, it is not known how the highly dynamic and variable perceptual signal is mapped to existing linguistic and semantic representations. In this novel approach, we utilized the natural acoustic variability of sounds and mapped them to magnetoencephalography (MEG) data using physiologically-inspired machine-learning models. We aimed at determining how well the models, differing in their representation of temporal information, serve to decode and reconstruct spoken words from MEG recordings in 16 healthy volunteers. We discovered that time-locking of the cortical activation to the unfolding speech input is crucial for the encoding of the acoustic-phonetic features. In contrast, time-locking was not highlighted in cortical processing of non-speech environmental sounds that conveyed the same meanings as the spoken words, including human-made sounds with temporal modulation content similar to speech. The amplitude envelope of the spoken words was particularly well reconstructed based on cortical evoked responses. Our results indicate that speech is encoded cortically with especially high temporal fidelity. This mechanism may contribute to the frequently reported entrainment of the cortical oscillations to the amplitude envelope of speech. Furthermore, the phoneme content was reflected in cortical evoked responses simultaneously with the spectrotemporal features, pointing to an instantaneous transformation of the unfolding acoustic features into linguistic representations during speech processing.\n\nSummary: Computational modeling of cortical responses to spoken words highlights the relevance of temporal tracking of spectrotemporal features, which is likely pivotal for transforming the acoustic-phonetic features into linguistic representations.

neuroscience↗

bioRxiv2018

Methods for computing the maximum performance of computational models of fMRI responses

Computational neuroimaging methods aim to predict brain responses (measured e.g. with functional magnetic resonance imaging [fMRI]) on the basis of stimulus features obtained through computational models. The accuracy of such prediction is used as an indicator of how well the model describes the computations underlying the brain function that is being considered. However, the prediction accuracy is bounded by the proportion of the variance of the brain response which is related to the measurement noise and not with the stimuli (or cognitive functions). The bound to the performance of a computational model to the prediction of brain responses has been referred to as the noise ceiling. In previous neuroimaging applications two methods have been proposed for estimating the noise ceiling based on either a split-half procedure or Monte Carlo simulations. These methods make different assumptions over the nature of the effects underlying the data, and, importantly, their relation has not been clarified yet. Here, we use a two-level generative framework to formally describe the partition between the variance of measurement noise and the stimulus related variance. In this framework we derive an analytical form for the noise ceiling that does not require computationally expensive simulations or a splitting procedure that reduce the amount of data. We describe the relation between the newly introduced noise ceiling estimator and the previous methods for variable levels of measurements noise using simulated data. Additionally, as the relation to the noise ceiling is used to make conclusions on the validity of a model with respect to others, we evaluate the effect the interplay between regularization (often used to estimate model fits to the data when the number of computational features in the model is large) and model complexity on the performance with respect to the noise ceiling. Finally, we show the differences between the methods on real fMRI data acquired at 7 Tesla. We demonstrate that while the split half estimator provides a pessimistic estimate of the noise ceiling due to the small amount of data available in conventional fMRI datasets, the parametric nature of the Monte Carlo estimator results in overly optimistic estimates. For this reason, for real data, we propose a robust procedure to the estimation of the noise ceiling based on bootstraps.\n\nAuthor SummaryEncoding computational models in brain responses measured with fMRI allows testing the algorithmic representations carried out by the neural population within voxels. The accuracy of a model in predicting new responses is used as a measure of the brain validity of this model, but the result of this analysis is determined not only by how precisely the model describes the responses but also by the quality of the data. In this article, we validate existing approaches to estimate the best possible accuracy that any computational model can achieve conditioned to the amount of measurement noise that is present in the experimental data (i.e. the noise ceiling). Additionally we introduce a close form estimation of the noise ceiling that does not require computationally or data expensive procedures. All the methods are compared using simulated and real fMRI data. We draw conclusions over the impact of regularisation procedures and model complexity and make practical recommendations on how to report the results of computational models in neuroimaging.

neuroscience↗