Evaluating Language Models for Biomedical Fact-Checking: A Benchmark Dataset for Cancer Variant Interpretation Verification

Abstract

Accurate interpretation of genomic variants is critical for precision oncology but remains slow and dependent on specialized expertise. Public knowledgebases such as the Clinical Interpretation of Variants in Cancer (CIViC) help by curating literature-backed variant interpretations in a structured form, yet verification and review have become major bottlenecks. Large language models (LLMs) offer a potential mechanism for accelerating biomedical claim verification, but their rapid turnover, variable availability, and known risks of unsupported reasoning require standardized and reproducible evaluation before integration into curation workflows. To address this, we developed CIViC-Fact, an expert-curated, full-text benchmark and evaluation framework. Domain experts linked structured cancer-variant claims to sentence-level evidence from source publications, including evidence from full-text articles, tables, and non-abstract sections that are commonly omitted from existing biomedical question-answering and scientific fact-checking datasets. Claim-verification reference labels were derived from CIViC records, revision histories and controlled data augmentation. A major finding of CIViC-Fact is that abstracts are insufficient for realistic biomedical claim verification. In the evaluated development subset of text-verifiable entries with full-text access, fewer than 30% could be fully validated from the abstract alone, highlighting the importance of full-text evaluation for biomedical curation. Upon the application of our fact-checking pipeline to newly submitted CIViC entries, after excluding entries requiring supplementary material or images for validation, automated retrieval successfully identified appropriate evidence for most cases (93%), supporting low-incremental-effort evaluation of future systems. Fine-tuning improved agreement with CIViC-Fact reference labels on the static benchmark, but larger general-purpose models performed better on a heterogeneous post-cutoff cohort. These findings support CIViC-Fact primarily as a reproducible framework for comparing evolving retrieval and verification systems rather than as validation of a single deployment-ready model. These findings suggest that, in a rapidly changing model landscape, the durable contribution is not a single optimized model but a reproducible benchmark framework that enables continual testing, model substitution, and lightweight updating through small high-quality few-shot exemplar sets.