Biology subjects

Nishimoto, S.

Publications and source records attributed to Nishimoto, S..

At least 19 recordsLinked to original sources

bioRxiv2026

Cross-domain encoding models reveal shared and domain-specific neural representations across language and mathematics

Whether language and mathematics rely on shared or distinct neural representations remains an unresolved question in cognitive neuroscience. Here we combine latent features from a large language model (LLM) with vertex-wise encoding models to examine cross-domain generalization between language and mathematics. Thirty-two participants performed sentence comprehension and calculation tasks during fMRI, and encoding models were trained using features embedded in a common latent space. Cross-domain prediction identified cortical regions associated with partially shared representations, most prominently the left 55b, while control analyses suggested that these effects could not be fully explained by low-level visual processing or simple task-general factors. Task-specificity contrasts revealed stronger language-related prediction in the left anterior superior temporal and angular gyri and math-related prediction in the left precentral and intraparietal sulci. Model-weight analyses further showed that shared and domain-specific prediction patterns were reflected in distinct weight profiles across cortical regions. Connectivity analyses showed task-dependent functional coupling between cross-domain regions and language- or math-related networks. Together, these findings suggest that language and mathematics involve partially shared neural representations alongside domain-specific cortical organization, helping reconcile previous contrasting views on their neural basis.

neuroscience↗

bioRxiv2026

A Quantitative Benchmark of Visual Information in Human Brain Recordings Across fMRI, MEG, and EEG

Noninvasive human brain recordings such as fMRI, MEG, and EEG are widely used to study visual representations, yet their relative information content has never been quantitatively benchmarked under matched experimental conditions. Here, we develop a unified decoding framework that enables fair cross-modal comparison using identical naturalistic stimuli (THINGS) and a common decoding analysis. Using a matched stimulus set, we first quantify modality performance via object-category decoding accuracy. Under these conditions, fMRI yielded the highest accuracy (mean 87%), MEG showed intermediate performance (mean 35%), and single-subject EEG achieved lower accuracy (mean 7%), while remaining reliably above chance level (1%). Notably, EEG performance improved substantially when decoding outputs were aggregated across participants in the form of a stimulus-level relational structure, yielding a group-level EEG measure (28%) with accuracy comparable to MEG. We then examined how decoding performance scales with the number of training stimuli and effective measurement time. These analyses revealed distinct efficiency-precision trade-offs across modalities: MEG and group-level EEG were most efficient for short measurement times, whereas fMRI achieved the highest asymptotic performance when sufficient data were collected. Together, these findings demonstrate that modality performance differs systematically even under tightly matched conditions and that decoding provides a principled way to assess the information content of fMRI, MEG, and EEG. The proposed unified framework offers practical guidelines for modality selection, experimental design, and data collection planning in neuroscience. Highlights{checkmark} Unified decoding framework enables quantitative, matched comparison of fMRI, MEG, and EEG. {checkmark}fMRI shows the highest decoding accuracy; MEG intermediate; EEG lower. {checkmark}Aggregating EEG decoding outputs into a stimulus level relational structure across subjects markedly improves performance to near MEG levels. {checkmark}MEG and EEG (aggregated across subjects) are most efficient for short recordings, whereas fMRI achieves the highest performance with longer measurement time.

neuroscience↗

bioRxiv2025

Second Language Proficiency Modulates Hierarchical Alignment between Brain Activity and Large Language Models

Second language (L2) comprehension is thought to proceed hierarchically, from basic word recognition to complex discourse-level understanding. However, the neural mechanisms underpinning this hierarchical progression remain poorly understood. Leveraging the hierarchical nature of linguistic representations in large language models (LLMs), we investigated how L2 proficiency modulates layer-wise brain-LLM alignment. Using functional magnetic resonance imaging (fMRI) data collected during discourse listening from 54 participants with varying levels of L2 proficiency, we constructed individualized encoding models to quantify how proficiency shapes the brain-LLM representational correspondence. In high-proficiency individuals, LLM-based models reliably predicted neural responses in a distributed network comprising temporal, frontal, and medial-parietal cortices, while for low-proficiency participants, prediction performance decreased. Notably, alignment in the superior temporal sulcus (STS) showed a robust correlation with L2 proficiency at mid-to-deep LLM layers, suggesting that the STS encodes higher-level linguistic abstractions essential for advanced language comprehension. These findings indicate that L2 proficiency modulates the alignment between LLMs and the brain in a graded, layer-dependent manner, providing insights into hierarchical neural language representations.

neuroscience↗

bioRxiv2025

Modulation of decodable semantic features of brain activity via selection attention

We frequently encounter linguistic information in multiple modalities, such as text and speech, simultaneously. In such cases, we understand the information by selectively attending to one of the modalities. Previous research has shown that selective attention to a specific stimulus modality modulates cortical activity patterns. However, it remains unclear which aspects of linguistic information are selectively modulated by attention and how such modulation influences the decodability of semantic content. To address this question, we constructed decoding models of latent semantic features from the functional magnetic resonance imaging data of six participants in both unimodal (either visual or auditory) and bimodal conditions (simultaneously visual and auditory, with participants attending to a single modality at a time). In unimodal conditions, we successfully decoded the semantic contents from the brain activity. Decodable features were consistent across modalities in both intra-modal and cross-modal decoding. For bimodal conditions, decoding accuracies for the attended stimuli were higher than for the ignored when training and test stimuli belonged to the same modality. Furthermore, decodable features were more consistent across modalities with attended than ignored stimuli in both intra-modal and cross-modal decoding. These results indicate common decodable semantic features regardless of the presentation modality and that selective attention enhances the semantic representations contributing to such decodability.

neuroscience↗

bioRxiv2024

Image retrieval based on closed-loop visual-semantic neural decoding

Neural decoding via the latent space of deep neural network models can infer perceived and imagined images from neural activities, even when the image is novel for the subject and decoder. Brain-computer interfaces (BCIs) using the latent space enable a subject to retrieve intended image from a large dataset on the basis of their neural activities but have not yet been realized. Here, we used neural decoding in a closed-loop condition to retrieve images of the instructed categories from 2.3 million images on the basis of the latent vector inferred from electrocorticographic signals of visual cortices. Using a latent space of contrastive language-image pretraining (CLIP) model, two subjects retrieved images with significant accuracy exceeding 80% for two instructions. In contrast, the image retrieval failed using the latent space of another model, AlexNet. In another task to imagine an image while viewing a different image, the imagery made the inferred latent vector significantly closer to the vector of the imagined category in the CLIP latent space but significantly further away in the AlexNet latent space, although the same electrocorticographic signals from nine subjects were decoded. Humans can retrieve the intended information via a closed-loop BCI with an appropriate latent space.

neuroscience↗

bioRxiv2024

Explicit description of viral capsid subunit shapes by unfolding dihedrons

Viral capsid assembly and the design of capsid-based nanocontainers critically depend on understanding the shapes and interfaces of constituent protein subunits. However, a comprehensive framework for characterizing these features is still lacking. Here, we introduce a novel approach based on spherical tiling theory that explicitly describes the 2D shapes and interfaces of subunits in icosahedral capsids. Our method unfolds spherical dihedrons defined by icosahedral symmetry axes, enabling systematic characterization of all possible subunit geometries. Applying this framework to real T = 1 capsid structures reveals distinct interface groups within this single classification, with variations in interaction patterns around 3-fold and 5-fold symmetry axes. We validate our classification through molecular docking simulations, demonstrating its consistency with physical subunit interactions. This analysis suggests different assembly pathways for capsid nucleation. Our general framework is applicable to other triangular numbers, paving the way for broader studies in structural virology and nanomaterial design.

biophysics↗

bioRxiv2024

Cell segmentation without annotation by unsupervised domain adaptation based on cooperative self-learning

Semantic cell segmentation from microscopic images is essential for the quantitative evaluation of cell morphology. Although supervised deep-learning-based models offer accurate segmentation, their performance degrades for unknown cell types. To address this problem, unsupervised domain adaptation methods based on adversarial training, self-training, or a combination of these approaches have been developed in recent years. These methods train the model using inference labels from the unknown domain as pseudo labels with reliability to resolve the discrepancy between the features of the unknown and known domains. However, conventional methods require a predefined threshold to calculate pseudo-labels reliability, leading to costly hyperparameter tuning. Here, we developed an unsupervised domain adaptation for semantic cell segmentation with cooperative self-learning (CULPICO: Cooperative Unsupervised Learning for PIxel-wise COloring) that does not require predefined threshold of the pseudo-labels reliability. The proposed method consists of two independent segmentation models and a mutual exchange mechanism of inference data. The models infer a label probability at each pixel and generate a pseudo-label as unsupervised learning. The pseudo-labels created by each model are mutually used as ground truth in the other model. Loss function is corrected by considering pixel-level discrepancies between the label probabilities inferred by the two models. The proposed method, despite being an unsupervised learning method, can segment efficiently the unknown cell types without labels with an accuracy comparable to supervised learning models. Our method, which could solve the performance degradation problem without constructing new datasets, is expected to accelerate life science by reducing the cost of extracting quantitative biological knowledge.

systems biology↗

bioRxiv2024

Asymmetric representation of symmetric semantic information in the human brain

Specific pairs of semantic entities have symmetric relationships, such as word pairs with opposite meanings (e.g., "intelligent" and "stupid"; "human" and "mechanical"). Such semantic symmetry is a key feature of semantic information. However, the representation of symmetric semantic information in the brain is not yet understood. Additionally, it is unclear whether symmetric pairs of semantic information do have symmetric representations in the brain? We addressed this question in a data-driven manner by using the voxelwise modeling of movie-evoked cortical response measured by functional magnetic resonance imaging. In this modeling, response in each voxel was predicted from semantic labels designated for each movie scene. The semantic labels consisted of 30 different items, including 15 pairs of semantically symmetric items. Each item was manually evaluated using a 5-point scale. By localizing the semantic representation associated with each item based on the voxelwise accuracy of brain-response predictions, we found that semantic representations of symmetric item pairs are broadly distributed but with little overlap in the cortex. Additionally, the weight of voxelwise models revealed highly complex, various patterns of cortical representations for each item pair. These results suggest that symmetric semantic information is rather asymmetric and heterogeneous representations in the human brain. Significance statementThis study aimed to investigate if symmetric pairs of semantic information have symmetric representations in the human brain using statistical modeling of functional magnetic resonance imaging signals evoked by naturalistic movies. We built a model based on movie labeling of symmetric semantic items to quantify the cortical representations of symmetric semantic entities. These findings showed that symmetric pairs of semantic entities are represented in widespread cortical regions; however, still exhibit little overlap in localization and heterogeneous representations. These results offer significant insights into the cortical representations of semantic symmetry and advance the understanding of representational structure of semantic information in the human brain.

neuroscience↗

bioRxiv2024

The Brain Tells a Story: Unveiling Distinct Representations of Semantic Content in Speech, Objects, and Stories in the Human Brain with Large Language Models

In recent studies, researchers have used large language models (LLMs) to explore semantic representations in the brain; however, they have typically assessed different levels of semantic content, such as speech, objects, and stories, separately. In this study, we recorded brain activity using functional magnetic resonance imaging (fMRI) while participants viewed 8.3 hours of dramas and movies. We annotated these stimuli at multiple semantic levels, which enabled us to extract latent representations of LLMs for this content. Our findings demonstrate that LLMs predict human brain activity more accurately than traditional language models, particularly for complex background stories. Furthermore, we identify distinct brain regions associated with different semantic representations, including multi-modal vision-semantic representations, which highlights the importance of modeling multi-level and multi-modal semantic representations simultaneously. We will make our fMRI dataset publicly available to facilitate further research on aligning LLMs with human brain function. Please check out our webpage at https://sites.google.com/view/llm-and-brain/.

neuroscience↗

bioRxiv2024

Cortical representational geometry of diverse tasks reveals subject-specific and subject-invariant cognitive structures

The variability in brain function forms the basis for our uniqueness. Prior studies indicate smaller individual differences and larger inter-subject correlation (ISC) in sensorimotor areas than in the association cortex. These studies, deriving information from brain activity, leave individual differences in cognitive structures based on task similarity relations unexplored. This study quantitatively evaluates these differences by integrating ISC, representational similarity analysis, and vertex-wise encoding models using functional magnetic resonance imaging across 25 cognitive tasks. ISC based on cognitive structures enables subject identification with 100% accuracy using at least 14 tasks. ISC is larger in the fronto-parietal association and higher-order visual cortices, suggesting subject-invariant cognitive structures in these regions. Principal component analysis reveals different cognitive structure configurations within these regions. This study provides new evidence of individual variability and similarity in abstract cognitive structures.

neuroscience↗

bioRxiv2023

Cortical representations of languages during natural dialogue

Individuals integrate their own words, their partners words, and the context of dialogue into coherent meanings. Research suggests that mutual understanding between speakers and listeners is supported by a shared representation between language production and comprehension1,2. However, it remains unknown how languages are represented in the brain during dialogue, which is characterized by dynamic, adaptive behaviours such as turn-taking3,4. Here, we used functional magnetic resonance imaging (fMRI) to compare language production and comprehension maps obtained from natural dialogue in the same participants to show that separate representations exist for language production and comprehension. While production showed selectivity towards the motor system and comprehension towards the auditory system, both production and comprehension were represented in broadly distributed regions. These separate representations were similar in amodal brain regions that integrate semantic5,6 and pragmatic information7,8, and provide a common ground for mutual understanding1,2, reflecting dynamic, complementary roles in interactive language use, including turn-taking3,4,9, backchannels10, and fillers11. Our findings suggest that separate and analogous linguistic representations for production and comprehension are interwoven in the same networks that underlie complementary interactions and making sense in dialogue.

neuroscience↗

bioRxiv2023

Localized and global computation for integrating prior value and sensory evidence in the mouse cerebral cortex

Adaptive behavior requires integrating prior knowledge of action outcomes and sensory evidence for making decisions while maintaining prior knowledge for future actions. As outcome- and sensory-based decisions are often tested separately, it is unclear how these processes are integrated in the brain. In a tone frequency discrimination task with two sound durations and asymmetric reward blocks, we found that neurons in the medial prefrontal cortex of male mice represented the additive combination of prior reward expectations and choices. The sensory inputs and choices were selectively decoded from the auditory cortex irrespective of reward priors and the secondary motor cortex, respectively, suggesting localized computations of task variables are required within single trials. In contrast, all the recorded regions represented prior values that needed to be maintained across trials. We propose localized and global computations of task variables in different time scales in the cerebral cortex.

neuroscience↗

bioRxiv2022

High-resolution image reconstruction with latent diffusion models from human brain activity

Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been employed for this task, reconstructing realistic images with high semantic fidelity is still a challenging problem. Here, we propose a new method based on a diffusion model (DM) to reconstruct images from human brain activity obtained via functional magnetic resonance imaging (fMRI). More specifically, we rely on a latent diffusion model (LDM) termed Stable Diffusion. This model reduces the computational cost of DMs, while preserving their high generative performance. We also characterize the inner mechanisms of the LDM by studying how its different components (such as the latent vector of image Z, conditioning inputs C, and different elements of the denoising U-Net) relate to distinct brain functions. We show that our proposed method can reconstruct high-resolution images with high fidelity in straight-forward fashion, without the need for any additional training and fine-tuning of complex deep-learning models. We also provide a quantitative interpretation of different LDM components from a neuroscientific perspective. Overall, our study proposes a promising method for reconstructing images from human brain activity, and provides a new framework for understanding DMs. Please check out our webpage at https://sites.google.com/view/stablediffusion-with-brain/.

neuroscience↗

bioRxiv2022

Artificial neural network modelling of the neural population code underlying mathematical operations

Mathematical operations have long been regarded as a sparse, symbolic process in neuroimaging studies. In contrast, advances in artificial neural networks (ANN) have enabled extracting distributed representations of mathematical operations. Recent neuroimaging studies have compared distributed representations of the visual, auditory and language domains in ANNs and biological neural networks (BNNs). However, such a relationship has not yet been examined in mathematics. Here we used the fMRI data of a series of mathematical problems with nine different combinations of operators to construct voxel-wise encoding models using both sparse operator and latent ANN features. Representational similarity analysis demonstrated shared representations between ANN and BNN, an effect particularly evident in the intraparietal sulcus. Feature-brain similarity analysis served to reconstruct a sparse representation of mathematical operations based on distributed ANN features. Such reconstruction was more efficient when using features from deeper ANN layers. Moreover, latent ANN features allowed the decoding of novel operators not used during model training from brain activity. The current study provides novel insights into the neural code underlying mathematical thought.

neuroscience↗

bioRxiv2022

Decoding Individual Differences in Mental Information from Human Brain Response Predicted by Convolutional Neural Networks

Recent advantages in brain decoding with functional magnetic resonance imaging (fMRI) have enabled us to estimate individual differences in mental information from brain responses to natural stimuli. However, the physical constraints and costs of fMRI measurements prevent brain decoding from achieving real-world applications. We address this issue by building a novel framework to decode individual differences in mental information under natural situations from brain responses predicted using convolutional neural networks (CNNs). Once the CNN-based prediction model is trained using measured response, mental information can be decoded from the predicted responses of individual brains with no need for additional fMRI measurements. The model was found to capture individual-difference patterns consistent with conventional decoding using measured responses in 81/87 items to be decoded. Our framework has great potential to decode personal mental information with low dependence on fMRI measurements, which could help substantially expand the applicability of brain decoding in daily life.

neuroscience↗

bioRxiv2022

Quantitative modeling demonstrates format-invariant representations of mathematical problems in the brain

Mathematical problems can be described in either symbolic form or natural language. Previous studies have reported that activation overlaps exist for these two types of mathematical problems, but it is unclear whether they are based on similar brain representations. Furthermore, quantitative modeling of mathematical problem solving has yet to be attempted. In the present study, subjects underwent 3 h of functional magnetic resonance experiments involving math word and math expression problems, and a read word condition without any calculations was used as a control. To evaluate the brain representations of mathematical problems quantitatively, we constructed voxel-wise encoding models. Both intra- and cross-format encoding modeling significantly predicted brain activity predominantly in the left intraparietal sulcus (IPS), even after subtraction of the control condition. Representational similarity analysis and principal component analysis revealed that mathematical problems with different formats had similar cortical organization in the IPS. These findings support the idea that mathematical problems are represented in the brain in a format-invariant manner.

neuroscience↗

bioRxiv2021

Preserved representations and decodability of diverse cognitive functions across the cortex, cerebellum, and subcortex

Which part of the brain contributes to our complex cognitive processes? Studies have revealed contributions of the cerebellum and subcortex to higher-order cognitive functions; however it is unclear whether such functional representations are preserved across the cortex, cerebellum, and subcortex. In this study, we used functional magnetic resonance imaging data with 103 cognitive tasks and constructed three voxel-wise encoding and decoding models independently using cortical, cerebellar, and subcortical voxels. Representational similarity analysis revealed that the structure of task representations is preserved across the three brain parts. Principal component analysis visualized distinct organizations of abstract cognitive functions in each part of the cerebellum and subcortex. More than 90% of the cognitive tasks were decodable from the cerebellum and subcortical activities, even for the novel tasks not included in model training. Furthermore, we discovered that the cerebellum and subcortex have sufficient information to reconstruct activity in the cerebral cortex.

neuroscience↗

bioRxiv2021

Modeling movie-evoked human brain activity using motion-energy and space-time vision transformer features

In this paper, the process of building a model for predicting human brain activity under video viewing conditions was described as a part of an entry into the Algonauts Project 2021 Challenge. The model was designed to predict brain activity measured using functional MRI (fMRI) by weighted linear summations of the spatiotemporal visual features that appear in the video stimuli (video features). Two types of video features were used: (1) motion-energy features designed based on neurophysiological findings, and (2) features derived from a space-time vision transformer (TimeSformer). To utilize the features of various video domains, the features of the TimeSformer models pre-trained using several different movie sets were combined. Through these model building and validation processes, results showed that there is a certain correspondence between the hierarchical representation of the TimeSformer model and the hierarchical representation of the visual system in the brain. The motion-energy features are effective in predicting brain activity in the early visual areas, while TimeSformer-derived features are effective in higher-order visual areas, and a hybrid model that uses motion energy and TimeSformer features is effective for predicting whole brain activity.

neuroscience↗