Differential reliance on sensory reinstatement and internally transformed representation during vivid retrieval of visual and auditory episodes

Abstract

Auditory memory is considered less detailed yet more durable than visual memory, implying a modality-specific memory retrieval process. We used fMRI and multivoxel pattern analyses to examine how 25 participants encoded and retrieved naturalistic sounds and videos. Both auditory and visual targets reinstated item-specific fine activation patterns in the association cortex during retrieval, and reinstatement strength correlates with subjective memory vividness. However, auditory episodes showed a markedly larger reliance on internally constructed representations than visual episodes, quantified by retrieval-retrieval similarity after removing encoding traces. Sensory reinstatement correlated more to the (detail-related) posterior hippocampus, while internal representations also correlated to the (gist-related) anterior hippocampus. Furthermore, temporal voice areas preserved gist-level (human versus non-human) information from encoding to retrieval, whereas fusiform face representations degraded. These findings reveal that auditory and visual memories share a common sensory reinstatement mechanism, but differ in the neural mechanism that supports retrieval, with participants favouring gist over perceptual details during auditory memory retrieval.