Biology subjects

Yamada, K. D.

Publications and source records attributed to Yamada, K. D..

7 recordsLinked to original sources

bioRxiv2023

Towards developing permanent memory system for artificial intelligence to acquire new knowledge after deployment

In this study, we developed a learning method for constructing a neural network system capable of memorizing data and recalling it without parameter updates. The system we built using this method is called the Appendable Memory system. The Appendable Memory system enables an artificial intelligence (AI) to acquire new knowledge even after deployment. It consists of two artificial intelligences: the Memorizer and the Recaller. This system is a key-value store built using neural networks. The Memorizer receives data and stores it in the Appendable Memory vector, which is dynamically updated when the AI acquires new knowledge. Meanwhile, the Recaller retrieves information from the Appendable Memory vector. What we want to make AI learn in this study is the operations of memorizing and recalling information. However, traditional machine learning methods make AI learn features inherent in the learning dataset. We demonstrate that the systems we intend to create cannot be realized by current machine learning methods, that is, by merely repeating the input and output learning sequences with AI. Instead, we propose a method to teach AI to learn operations, by completely removing the features contained in the learning dataset. Specifically, we probabilized all the data involved in learning. This measure prevented AI from learning the features of the data. The learning method proposed in the study differs from traditional machine learning methods and provides fundamental approaches for building an AI system that can store information in a finite memory and recall it at a later date.

neuroscience↗

bioRxiv2022

Relation is an option for processing context information

Attention mechanisms are one of the most frequently used architectures in the development of artificial intelligence because they can process contextual information efficiently. Various artificial intelligence mechanisms such as transformer for processing natural language, image data, etc. include the attention mechanism. Since attention is a powerful component to realize artificial intelligence, various improvements have been made to enhance its performance. The time complexity of attention depends on the square of input sequence length. Methods to improve time complexity of attention are one of the hottest topics in various studies. In this study, we have devised a mechanism called relation that can understand the context information of sequential data without using attention. Relation is very simple to implement and its time complexity depends only on the length of the sequences; a comparison of the performance of neural networks with relation and attention mechanisms on several benchmark datasets shows that relation achieved the same context processing capability as attention with less computation time. Hence, relation is an ideal option for processing context information.

neuroscience↗

bioRxiv2021

Prediction and Characterization of Disorder-Order Transition Regions in Proteins by Deep Learning

The biological functions of proteins are traditionally thought to depend on well-defined three-dimensional structures, but many experimental studies have shown that disordered regions lacking fixed three-dimensional structures also have crucial biological roles. In some of these regions, disorder-order transitions are also involved in various biological processes, such as protein-protein interaction and ligand binding. Therefore, it is crucial to study disordered regions and structural transitions for further understanding of protein functions and folding. Owing to the costs and time requirements of experimental identification of natively disordered or transitional regions, the development of effective computational methods is a key research goal. In this study, we used overall residue dependencies and deep representation learning for prediction and reused the obtained disordered regions for the prediction of disorder-order transitions. Two similar and related prediction tasks were combined. Firstly, we developed a novel deep learning method, Res-BiLstm, for residue-wise disordered region prediction. Our method outperformed other predictors with respect to almost all criteria, as evaluated using an independent test set. For disorder-order transition prediction, we proposed a transfer learning method, Res-BiLstm-NN, with an acceptable but unbalanced performance, yielding reasonable results. To grasp underlining biophysical principles of disorder-order transitions, we performed qualitative analyses on the obtained results and discovered that most transitions have strong disordered or ordered preferences, and more transitions are consistent with the ordered state than the disordered state, different from conventional wisdom. To the best of our knowledge, this is the first sizable-scale study of transition prediction. Availabilityhttps://github.com/Yanzziang/Transition_Disorder_Prediction Contactkengo@ecei.tohoku.ac.jp

bioinformatics↗

bioRxiv2020

Developing a novel recurrent neural network architecture with fewer parameters and good learning performance

Recurrent neural networks (RNNs) are among the most promising of the many artificial intelligence techniques now under development, showing great potential for memory, interaction, and linguistic understanding. Among the more sophisticated RNNs are long short-term memory (LSTM) and gated recurrent units (GRUs), which emulate animal brain behavior; these methods yield superior memory and learning speed because of the excellent core structure of their architectures. In this study, we attempted to make further improvements in core structure and develop a novel, compact architecture with a high learning speed. We stochastically generated 30000 RNN architectures, evaluated their performance, and selected the one most capable of memorizing long contexts with relatively few parameters. This RNN, YamRNN, had fewer parameters than LSTM and GRU by a factor of two-thirds or better and reduced the time required to achieve the same learning performance on a sequence classification task as LSTM and GRU by 80% at maximum. This novel RNN architecture is expected to be useful for addressing problems such as predictions and analyses on contextual data and also suggests that there is room for the development of better architectures.

neuroscience↗

bioRxiv2018

Hyperparameter-free optimizer of gradient-descent method incorporating unit correction and moment estimation

In the deep learning era, stochastic gradient descent is the most common method used for optimizing neural network parameters. Among the various mathematical optimization methods, the gradient descent method is the most naive. Adjustment of learning rate is necessary for quick convergence, which is normally done manually with gradient descent. Many optimizers have been developed to control the learning rate and increase convergence speed. Generally, these optimizers adjust the learning rate automatically in response to learning status. These optimizers were gradually improved by incorporating the effective aspects of earlier methods. In this study, we developed a new optimizer: YamAdam. Our optimizer is based on Adam, which utilizes the first and second moments of previous gradients. In addition to the moment estimation system, we incorporated an advantageous part of AdaDelta, namely a unit correction system, into YamAdam. According to benchmark tests on some common datasets, our optimizer showed similar or faster convergent performance compared to the existing methods. YamAdam is an option as an alternative optimizer for deep learning.

bioinformatics↗

bioRxiv2017

De novo profile generation based on sequence context specificity with the long short-term memory network

Long short-term memory (LSTM) is one of the most attractive deep learning methods to learn time series or contexts of input data. Increasing studies, including biological sequence analyses in bioinformatics, utilize this architecture. Amino acid sequence profiles are widely used for bioinformatics studies, such as sequence similarity searches, multiple alignments, and evolutionary analyses. Currently, many biological sequences are becoming available, and the rapidly increasing amount of sequence data emphasizes the importance of scalable generators of amino acid sequence profiles. We employed the LSTM network and developed a novel profile generator to construct profiles without any assumptions, except for input sequence context. Our method could generate better profiles than existing de novo profile generators, including CSBuild and RPS-BLAST, on the basis of profile-sequence similarity search performance with linear calculation costs against input sequence size. In addition, we analyzed the effects of the memory power of LSTM and found that LSTM had high potential power to detect long-range interactions between amino acids, as in the case of beta-strand formation, which has been a difficult problem in protein bioinformatics using sequence information. We demonstrated the importance of sequence context and the feasibility of LSTM on biological sequence analyses. Our results demonstrated the effectiveness of memories in LSTM and showed that our de novo profile generator, SPBuild, achieved higher performance than that of existing methods for profile prediction of beta-strands, where long-range interactions of amino acids are important and are known to be difficult for the existing window-based prediction methods. Our findings will be useful for the development of other prediction methods related to biological sequences by machine learning methods.

bioinformatics↗

bioRxiv2017

Optimizing scoring function of dynamic programming of pairwise profile alignment using derivative free neural network

A profile comparison method with position-specific scoring matrix (PSSM) is one of the most accurate alignment methods. Currently, cosine similarity and correlation coefficient are used as scoring functions of dynamic programming to calculate similarity between PSSMs. However, it is unclear that these functions are optimal for profile alignment methods. At least, by definition, these functions cannot capture non-linear relationships between profiles. Therefore, in this study, we attempted to discover a novel scoring function, which was more suitable for the profile comparison method than the existing ones. Firstly we implemented a new derivative free neural network by combining the conventional neural network with evolutionary strategy optimization method. Next, using the framework, the scoring function was optimized for aligning remote sequence pairs. Nepal, the pairwise profile aligner with the novel scoring function significantly improved both alignment sensitivity and precision, compared to aligners with the existing functions. Nepal improved alignment quality because of adaptation to remote sequence alignment and increasing the expressive power of similarity score. The novel scoring function can be realized using a simple matrix operation and easily incorporated into other aligners. With our scoring function, the performance of homology detection and/or multiple sequence alignment for remote homologous sequences would be further improved.

bioinformatics↗