Ellington, A. O. | bioRxiv Science

bioRxiv2026

Overestimating zero-shot fitness prediction: Broad benchmarks mask local failures and practical limitations

Deep learning models have emerged as promising tools in protein engineering. In particular, they can be used to predict mutation fitness without the need for task-specific training, a process known as zero-shot prediction. However, the respective strengths and limitations of zero-shot predictions remain poorly understood. Here, we argue that commonly used large-scale benchmarks obscure important failure modes relevant to practical protein engineering, including an inability to pinpoint highly fit mutations or variants driving new-to-nature functions. Beyond these practical failures, we identify a fundamental limitation of zero-shot prediction: a generic fitness score cannot simultaneously optimize for distinct, competing engineering targets, meaning it is inherently disconnected from the phenotype of interest. Moreover, in a systematic comparison of a wide range of available models, we demonstrate that most models show comparable zero-shot performance, irrespective of model architecture and/or input modality (sequence vs. structure). Ultimately, we find that zero-shot predictions serve only as coarse filters separating fit mutations from deleterious ones, failing to reliably identify the mutations that would be most valuable in protein engineering.

bioengineering↗

bioRxiv2023

Sequence and structure based deep learning models represent different aspects of protein biochemistry

Deep learning models are seeing increased use as methods to predict mutational effects or allowed mutations in proteins. The models commonly used for these purposes include large language models (LLMs) and 3D Convolutional Neural Networks (CNNs). These two model types have very different architectures and are commonly trained on different representations of proteins. LLMs make use of the transformer architecture and are trained purely on protein sequences whereas 3D CNNs are trained on voxelized representations of local protein structure. While comparable overall prediction accuracies have been reported for both types of models, it is not known to what extent these models make comparable specific predictions and/or generalize protein biochemistry in similar ways. Here, we perform a systematic comparison of two LLMs and two structure-based models (CNNs) and show that the different model types have distinct strengths and weaknesses. The overall prediction accuracies are largely uncorrelated between the sequence- and structure-based models. Overall, the two structure-based models are better at predicting buried aliphatic and hydrophobic residues whereas the two LLMs are better at predicting solvent-exposed polar and charged amino acids. Finally, we find that a combined model that takes the individual model predictions as input can leverage these individual model strengths and results in significantly improved overall prediction accuracy.

biochemistry↗