bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.64898/2026.04.18.719404

Benchmarking Agentic Large Language Models for ComplexProtein-Set Functional Annotation

Zhang, X.
View original source ↗Export metadata ↓

Abstract

Large language model (LLM) agents are increasingly used to synthesize heterogeneous bioinformatics evidence, but their reliability for high-volume biological annotation remains poorly characterized. We evaluated three agent configurations on a controlled protein annotation task: Claude App with Claude Opus 4.7, Claude Code CLI with Claude Opus 4.7 and Claude Scientific Skills, and Codex App with GPT-5.4 and Claude Scientific Skills. Each configuration was run three times on the same verbatim prompt, the same 73 selected orthogroup FASTA files (1,705 protein sequences), and the same local evidence: Swiss-Prot BLASTP output, Pfam/HMMER domain hits, DeepTMHMM topology predictions, and SignalP secretion predictions. We audited the nine outputs for coverage, biological correctness, missing evidence, hallucinated or over-specific annotations, and within-method consistency, then merged the best-supported evidence into a final orthogroup annotation table. All nine runs covered all 73 orthogroups, indicating that the agents could retrieve and organize the complete input set. However, normalized calcification-relevance calls were only moderately reproducible: within-method exact tier agreement ranged from 0.397 to 0.685 for Claude App (mean 0.562), 0.342 to 0.740 for Claude Code (mean 0.516), and 0.411 to 0.630 for Codex App (mean 0.539), and the per-run number of high-confidence calls varied from 0 to 12 across the nine runs. The final curated table retained 3 high-confidence, 9 moderate, 18 watchlist, and 43 low-relevance orthogroups. The most robust direct candidates were sulfatase (OG0017138) and sulfotransferase (OG0020703) families and an FG-GAP/integrin-like surface protein family (OG0018986), whereas common error modes included elevating pentapeptide-repeat orthogroups on motif evidence alone, treating weakly secreted housekeeping enzymes as matrix proteins, and taking low-complexity BLAST labels at face value. Skill-enabled agents improved file handling, evidence traceability, and reproducibility of computational checking, but they did not eliminate biological overinterpretation. These results support a best-practice workflow in which LLM agents draft annotations only after deterministic evidence tables are generated, with explicit scoring rules, provenance columns, run-to-run replication, and expert review of high-impact claims.

Explore related subjects

bioinformaticsPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zhang, X.. 2026-04-21. Benchmarking Agentic Large Language Models for ComplexProtein-Set Functional Annotation. https://doi.org/10.64898/2026.04.18.719404

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Heterogeneous Graph Contrastive Learning for Drug-Gene-Disease Motif Prediction

Drug repurposing and target discovery offer critical strategies for advancing therapeutic development by uncovering the potential biological pathways and novel associations among drugs, genes, and diseases. However, experimental discovery remains expensive and time-consuming, which limits the scalability of large-scale studies. In addition, existing computational approaches often struggle to effectively integrate heterogeneous biomedical data, capture the complex higher-order topological signatures of biological interactomes, and generalize to unseen entities. Here, we present HANAMI (Heterogeneous grAph coNtrastive leArning for drug-gene-disease Motif predIction), a multi-view deep graph learning framework designed to model complex interactions among drugs, genes, and diseases. HANAMI integrates diverse heterogeneous biomedical knowledge, including chemical structures, genomic sequences, and clinical phenotypes, and leverages relation-aware topology encoding, structure-aware aggregation, and contrastive learning to enable accurate motif prediction with biological context from the network. Systematic evaluation on benchmark datasets shows that HANAMI achieves up to 6% improvements over existing state-of-the-art methods in predicting drug-gene-disease motifs. The framework further demonstrates strong inductive generalization, maintaining an [~]18% performance advantage in zero-shot settings involving previously unseen entities. Beyond predictive performance, HANAMI effectively prioritizes drug-disease relationships investigated in Phase II or III trials while identifying candidate genes that suggest plausible mechanistic links. Together, HANAMI provides a computational framework for interpreting complex biomedical interactions, offering a scalable foundation to accelerate drug repurposing and therapeutic innovation.

bioinformatics↗
bioRxiv2026

PTMExplorer: A Multi-Dimensional Integrative Visualization Platform for Protein Post-Translational Modification Function and Structure

Deciphering the functions of post-translational modifications (PTMs) is a critical bridge connecting large-scale modification proteomics data to mechanistic studies. However, most existing tools for visualizing PTM omics data are limited to site catalogs or single-dimensional feature displays. They lack the capability to simultaneously map user-derived differential modification sites onto multi-dimensional contexts, including protein three-dimensional (3D) structure, evolutionary conservation, functional sites, and disease associations. This limitation makes it difficult for researchers to rapidly assess the biological importance of candidate sites from among a vast number of differentially modified sites. Here, we present PTMExplorer, an interactive platform for the multi-dimensional visualization of protein PTMs. PTMExplorer comprises three core modules: PTM Inspector, built upon ProtVista, provides a multi-track, sequence-feature integrated view incorporating intrinsically disordered region (IDR) prediction (via flDPnn), surface accessibility calculation (via FreeSASA), and UniProt functional annotations; PTM 3D Locator, leveraging the Nightingale/Mol* engine, anchors modification sites onto AlphaFold/Protein Data Bank (PDB) 3D structures through residue mapping via PDBe-SIFTS; and PTM Overview, utilizing the R circlize package, presents a panoramic polar circos plot illustrating modification distribution and inter-group differential regulation. Additionally, three major disease-associated modification databases (PTMD, qPTM, and PhosCancer) are integrated as PTM-Disease Nexus, enabling co-localization comparison between user-defined differential sites and reported disease-related sites. PTMExplorer currently supports eight model organisms, accepts user-uploaded differential analysis results, and provides multi-dimensional annotations and various visualization options (https://www.bioladder.cn/PTMExplorer/). Using a multi-omics dataset from hepatocellular carcinoma (18 patients, 9 modification types) as a case study, we demonstrate the practical utility of PTMExplorer in screening potential biomarkers, revealing multi-modification coordination mechanisms, and distinguishing between absolute and relative quantification patterns.

bioinformatics↗
bioRxiv2026

Integrative analysis of the MDM2 promoter switch and cellular lineage plasticity in colorectal cancer: a contrast between the autonomous-proliferation type (CIN/CMS2) and the environment-adaptive type (MSI/gastric metaplasia)

Background: Biomarkers that stratify colorectal cancer (CRC) by therapeutic responsiveness and are measurable directly in biopsy specimens remain insufficiently established. We investigated whether usage of the dual MDM2 promoters (P1/P2) acts as a molecular switch separating two diametrically opposed CRC phenotypes: a chromosomal instability type and an environment adaptive type (microsatellite instability/serrated pathway with gastric metaplasia). Methods: Sixty three organoid samples from 22 patients with CRC were classified morphologically by deep learning (VGG16) and molecularly by an MDM2 Splicing Index derived from expression arrays. The P1 and P2 signatures (gene sets characterizing P1 and P2dominant samples) were externally validated in TCGA-COAD/READ (n = 624) and GSE39582 (n = 536), 1,160 cases in total, and therapeutic implications were tested in public cell line panels (GDSC2, DepMap) and in 65 lines of an independent patient derived CRC organoid biobank. Results: Deep learning morphological classification reached 98.5% test accuracy (64/65), and morphology corresponded to P1/P2 isoform usage: Type1 (compact glandular) morphology predominated in P1 dominant samples (median Type1 fraction 0.826 versus 0.444) and non Type1 (cystic mucinous) morphology in P2 dominant samples (AUC 0.79). Both signatures differed across the four consensus molecular subtypes , and the P2 signature was higher in mismatch repair deficient (microsatellite-unstable) tumors. Promoter usage quantified directly (P2_index) was higher in TP53 wild type tumors , consistent with P2 being p53-inducible. TP53 wild type cell lines were more sensitive to the MDM2 inhibitor Nutlin 3a and were more dependent on MDM2 in the DepMap CRISPR screen ; among 198 GDSC2 drugs, Nutlin 3a correlated most strongly with the P2 score. In the independent biobank, TP53 wild type lines (17) were more sensitive to nutlin-3 than mutant lines (48) (median log(IC50) 1.386 versus 4.283). Conclusions: MDM2 promoter choice (P1/P2) co-varies with the lineage identity of cancer cells and with the secretory, mucin rich character of the tumor tissue, consistent with a molecular-switch role alongside p53 suppression. The MDM2 P1/P2 ratio, measurable by RT qPCR or targeted NGS, is a candidate molecular-classification and therapeutic-stratification biomarker corresponding robustly to CMS, MSI, and TP53 mutation status.

bioinformatics↗