Lisso, F. | bioRxiv Science

bioRxiv2026

An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph

We combine GeneNetwork's 1TB structured database with its metadata as an RDF graph and an associated literature corpus. We benchmark frontier and local LLMs querying it through four architectures: document, graph and agentic retrieval-augmented generation (RAG), and a combination of the three. We evaluate them on 40 domain specific expert-curated questions with a new metric: the satisfaction frequency psi (psi, i.e, the share of questions answered with precision and recall of at least 70% or an f1 of at least 60%). psi ranks systems the same way precision, recall and f1 -- a machine learning metric built from the harmonic mean of precision and recall -- but is easier to read and assess for humans. System architecture had a large effect: with full use of Claude Opus 4.8 the plain document RAG of our earlier GNQA system had a psi of 36% against 86% for our new combined tools. Delegating most of the work from Claude Opus 4.8 to a local model halved query costs while still reaching a psi of 59%.

bioinformatics↗

bioRxiv2024

Creating a biomedical knowledge base by addressing GPT inaccurate responses and benchmarking context

We created GNQA, a generative pre-trained transformer (GPT) knowledge base driven by a performant retrieval augmented generation (RAG) with a focus on aging, dementia, Alzheimers and diabetes. We uploaded a corpus of three thousand peer reviewed publications on these topics into the RAG. To address concerns about inaccurate responses and GPT hallucinations, we implemented a context provenance tracking mechanism that enables researchers to validate responses against the original material and to get references to the original papers. To assess the effectiveness of contextual information we collected evaluations and feedback from both domain expert users and citizen scientists on the relevance of GPT responses. A key innovation of our study is automated evaluation by way of a RAG assessment system (RAGAS). RAGAS combines human expert assessment with AI-driven evaluation to measure the effectiveness of RAG systems. When evaluating the responses to their questions, human respondents give a "thumbs-up" 76% of the time. Meanwhile, RAGAS scores 90% on answer relevance on questions posed by experts. And when GPT-generates questions, RAGAS scores 74% on answer relevance. With RAGAS we created a benchmark that can be used to continuously assess the performance of our knowledge base. Full GNQA functionality is embedded in the free GeneNetwork.org web service, an open-source system containing over 25 years of experimental data on model organisms and human. The code developed for this study is published under a free and open-source software license at https://git.genenetwork.org/gn-ai/tree/README.md.

scientific communication and education↗