Robust Decoding of Speech Acoustics from EEG: Going Beyond the Amplitude Envelope

Abstract

ObjectiveDuring speech perception, properties of the acoustic stimulus can be reconstructed from the listeners brain using methods such as electroencephalography (EEG). Most studies employ the amplitude envelope as a target for decoding; however, speech acoustics can be characterised on multiple dimensions, including as spectral descriptors. The current study assesses how robustly an extended acoustic feature set can be decoded from EEG under varying levels of intelligibility and acoustic clarity. ApproachAnalysis was conducted using EEG from 38 young adults who heard intelligible and non-intelligible speech that was either unprocessed or spectrally degraded using vocoding. We extracted a set of acoustic features which, alongside the envelope, characterised instantaneous properties of the speech spectrum (e.g., spectral slope) or spectral change over time (e.g., spectral flux). We establish the robustness of feature decoding by employing multiple model architectures and, in the case of linear decoders, by standardising decoding accuracy (Pearsons r) using randomly permuted surrogate data. Main resultsLinear models yielded the highest r relative to non-linear models. However, the separate decoder architectures produced a similar pattern of results across features and experimental conditions. After converting r values to Z-scores scaled by random data, we observed substantive differences in the noise floor between features. Decoding accuracy significantly varies by spectral degradation and speech intelligibility for some features, but such differences are reduced in the most robustly decoded features. This suggests acoustic feature reconstruction is primarily driven by generalised auditory processing. SignificanceOur results demonstrate that linear decoders perform comparably to non-linear decoders in capturing the EEG response to speech acoustic properties beyond the amplitude envelope, with the reconstructive accuracy of some features also associated with understanding and spectral clarity. This sheds light on how sound properties are differentially represented by the brain and shows potential for clinical applications moving forward.