In Other Words: Learning When to Trust Generated Protein Function Descriptions

Abstract

Generative protein-to-text models write function descriptions in free-form English but give no indication of which to believe. Indeed, free text is hard to evaluate. We introduce a large language model judge of equivalence of predicted vs. ''true'' function descriptions, validated against blinded expert curation. We also present ProtTale, our sequence-to-text model, that learns from the judge's verdicts to estimate its own reliability, pJay. On unseen Swiss-Prot enzymes, ProtTale and three other leading methods tie on accuracy but are right about different proteins. Here, ProtTale's unique correct predictions outnumber all other methods' combined. Scored from sequence alone, pJay is a plug-in filter, raising all models' performance. On short proteins, it raises precision for every tested method nearly twofold. We argue that generative annotation becomes useful not by being right more often, but by knowing when it can be trusted---a standard that every generative model in biology should meet.