Diagnosing protein sequence search in the era of language models

Abstract

Protein language model (PLM) based search is rapidly emerging as a successor to classical sequence alignment, with recent high-profile studies reporting substantial improvements in speed and remote homology detection. However, success on standard benchmarks does not guarantee that similarity derived from PLM embeddings constitutes reliable biological evidence. Here, we introduce PLM-GUARD, a diagnostic framework designed to interrogate the underlying meaning of protein search scores and assess their biological trustworthiness. PLM-GUARD comprises six sanity checks spanning biological fidelity, semantic validity, and manipulation safety. Across eight representative search methods, classical alignment-based systems demonstrate remarkable robustness, whereas current PLM-based methods fail broadly across all three dimensions. Notably, hybrid methods show intermediate results, indicating that alignment is still critical for ensuring biologically grounded correspondence. Our findings provide a timely clarification for the field and underscore the necessity of diagnostic evaluation as protein search enters the era of language models.