From Images and Words to Modality-Independent Semantic Representations: When and Where They Emerge

Abstract

Conceptual knowledge can be accessed from different modalities, such as seeing an image or reading a written word. While prior neuroimaging studies implicate anterior temporal cortex in modality-independent semantic representations, when and how representations evoked by images and words converge onto a modality-independent code remains unclear. Moreover, evidence for modality-independent category representations comes largely from tasks that explicitly required semantic processing, leaving open whether such representations emerge when semantic category and stimulus modality are task-irrelevant. We magnetoencephalography (MEG) to determine when and where modality-independent semantic category representations emerge across images and words when participants (n = 32) encoded six-item sequences containing an exemplar of six semantic categories counterbalanced across positions and reported the ordinal position of a probed item. Decoding performance was above chance for all categories in each modality from approximately 70 ms relative to item presentation. Generalization peaked at similar latencies for the two modalities (~520-530 ms), where image and word classifiers had the most similar coefficients. In both transfer directions, activity elicited by images generalized mostly to later activity elicited by words, and a given period of the image response shared its category code with a longer period of the word response. Source-space decoding around the peak of generalization identified the same regions for both transfer directions: left and right anterior temporal lobe, perisylvian cortex (more extensive in the left hemisphere), right visual cortex, and bilateral orbitofrontal cortex, indicating a modality-independent category code distributed across the cortex.