Fine-tuning Protein Embeddings for GeneralizableAnnotation Propagation

Abstract

A central goal of bioinformatics research is to understand proteins on a functional level, typically by extrapolating from experimental results with the protein sequence information. One strategy is to assume that proteins with similar sequences will also share function. This has the benefit of being interpretable; it gives a very clear idea of why a protein might have a particular function by comparing with the most similar reference example. However, direct machine learning classifiers now outperform pure sequence similarity methods in raw prediction ability. A hybrid method is to use pre-trained language models to create protein embeddings, and then indirectly predict protein function using their relative similarity. We find that fine-tuning an auxiliary objective on protein function indirectly improves these hy-brid methods, to the point that they are in some cases better than direct classifiers. Our empirical results demonstrate that interpretable protein comparison models can be developed using fine-tuning techniques, without cost, or even with some benefit, to overall performance. K-nearest neighbors (KNN) embedding-based models also offer free generalization to previously unknown classes, while continuing to outperform only pre-trained models, further demonstrating the potential of fine-tuned embeddings outside of direct classification. Significance StatementFor the purposes of biological understanding, or engineering, it is particularly important that protein annotation is done through interpretable means, such as comparison to existing proteins with known properties. Our work builds upon previous efforts to do so by embedding proteins into high-dimensional vectors with pre-trained protein language models. We demonstrate that further training of these models on protein function classification drastically improves their performance, while allowing for exactly the same similarity based classifications on both known and novel potential functions.