bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.64898/2026.07.26.740845

Vision-Language Models for Image-Based Dietary Assessment: A Benchmark of Accuracy, Cost, and Prompt Strategies Across Ten Models

Sterling, S.·Berube, L. T.·Glenn, A. J.·Shaukat, A.·Barua, S.·Grams, M. E.·Tsirigos, A.
View original source ↗Export metadata ↓

Abstract

BackgroundDietary assessment is the cornerstone of clinical management and research studies evaluating diet and health. Traditional methods such as food diaries and 24-hour recalls can be burdensome, prone to recall bias, and difficult to adhere to. Image-based dietary assessment using vision-language models (VLMs) offers a potential solution. ObjectiveOur goal was to benchmark state-of-the-art VLMs for automated food recognition, weight estimation, and calorie estimation using Googles Nutrition5k dataset. MethodsWe evaluated 3,229 food images using ten approaches: proprietary VLMs (Gemini 2.0 Flash, 2.5 Flash, 3.0 Flash, and 3.1 Flash Lite; GPT- 4o, GPT-4o-mini, and GPT-5 Mini; and Claude Haiku 4.5), an open-source VLM (Qwen2-VL-7B), and a commercial food recognition API (FatSecret). We assessed calorie and weight estimation using Lins Concordance Correlation Coefficient (CCC) and component detection using Jaccard similarity. ResultsGemini 3.0 Flash achieved the best calorie estimation (CCC 0.767, MAE 80.7 kcal), while Gemini 3.1 Flash Lite offered very comparable accuracy (CCC 0.754) with the highest ingredient recognition (Jaccard 0.655) at the lowest cost among top-performing models ($0.59/1K images). Among earlier-generation models, Gemini 2.0 Flash remained competitive (CCC 0.742, Jaccard 0.621) at a fraction of the cost ($0.10/1K images). A human validation study in which four annotators reviewed 440 images revealed systematic omissions in the original Nutrition5k labels. After correction, the extrapolated ingredient-overlap score for Gemini 2.0 Flash increased from 0.62 to an estimated 0.82, suggesting that raw Jaccard scores substantially underestimate true model performance. ConclusionsCurrent VLMs can perform automated dietary assessment with reasonable accuracy from single overhead photographs. Our results inform model selection for dietary assessment applications and highlight remaining challenges in calorie estimation and component detection for complex, multi-item meals.

Explore related subjects

bioinformaticsPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sterling, S., Berube, L. T., Glenn, A. J., Shaukat, A., Barua, S., Grams, M. E., Tsirigos, A.. 2026-07-29. Vision-Language Models for Image-Based Dietary Assessment: A Benchmark of Accuracy, Cost, and Prompt Strategies Across Ten Models. https://doi.org/10.64898/2026.07.26.740845

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

A meta-interaction basis for cell-cell communication in tissues

Tissue function depends on signals exchanged between cells and the responses they elicit. Yet whether diverse cell-cell interactions in situ form recurrent sender-receiver programs remains unclear. We present SpiderNet, an interpretable representation-learning framework that discovers such directed programs as a compact basis of cell-cell meta-interactions (MIs) from spatial transcriptomics. SpiderNet jointly learns which sender regulators, ligand-receptor pairs, and receiver targets define each MI and where each program is active across neighboring cell pairs. The resulting representation traces multicellular relays and links communication to cell states, perturbation responses, and phenotypes. SpiderNet recovers ground-truth MIs and their molecular components in simulations and, in real tissues, shows stronger direction-specific agreement with independently curated regulatory programs in senders and receivers than alternative methods. Across more than 5.8 million spatially profiled cells, SpiderNet resolves an SPP1-THBS relay linking monocytes, fibroblasts, and tumor cells within an immune-suppressive ovarian cancer niche, predicts T-cell responses to held-out melanoma-cell perturbations, and identifies a T-cell-associated brain-aging program and age-predictive signals that transfer across regions and platforms. It reveals a recurrent pan-cancer COLLAGEN-linked fibroblast-tumor program whose projected abundance in independent cohorts is associated with poorer survival and non-response to immunotherapy. SpiderNet thus establishes MIs as a reusable organizational layer between molecular interactions and tissue phenotypes, providing a framework to resolve, compare, trace, and perturb multicellular regulation in situ.

bioinformatics↗
bioRxiv2026

Heterogeneous Graph Contrastive Learning for Drug-Gene-Disease Motif Prediction

Drug repurposing and target discovery offer critical strategies for advancing therapeutic development by uncovering the potential biological pathways and novel associations among drugs, genes, and diseases. However, experimental discovery remains expensive and time-consuming, which limits the scalability of large-scale studies. In addition, existing computational approaches often struggle to effectively integrate heterogeneous biomedical data, capture the complex higher-order topological signatures of biological interactomes, and generalize to unseen entities. Here, we present HANAMI (Heterogeneous grAph coNtrastive leArning for drug-gene-disease Motif predIction), a multi-view deep graph learning framework designed to model complex interactions among drugs, genes, and diseases. HANAMI integrates diverse heterogeneous biomedical knowledge, including chemical structures, genomic sequences, and clinical phenotypes, and leverages relation-aware topology encoding, structure-aware aggregation, and contrastive learning to enable accurate motif prediction with biological context from the network. Systematic evaluation on benchmark datasets shows that HANAMI achieves up to 6% improvements over existing state-of-the-art methods in predicting drug-gene-disease motifs. The framework further demonstrates strong inductive generalization, maintaining an [~]18% performance advantage in zero-shot settings involving previously unseen entities. Beyond predictive performance, HANAMI effectively prioritizes drug-disease relationships investigated in Phase II or III trials while identifying candidate genes that suggest plausible mechanistic links. Together, HANAMI provides a computational framework for interpreting complex biomedical interactions, offering a scalable foundation to accelerate drug repurposing and therapeutic innovation.

bioinformatics↗
bioRxiv2026

PTMExplorer: A Multi-Dimensional Integrative Visualization Platform for Protein Post-Translational Modification Function and Structure

Deciphering the functions of post-translational modifications (PTMs) is a critical bridge connecting large-scale modification proteomics data to mechanistic studies. However, most existing tools for visualizing PTM omics data are limited to site catalogs or single-dimensional feature displays. They lack the capability to simultaneously map user-derived differential modification sites onto multi-dimensional contexts, including protein three-dimensional (3D) structure, evolutionary conservation, functional sites, and disease associations. This limitation makes it difficult for researchers to rapidly assess the biological importance of candidate sites from among a vast number of differentially modified sites. Here, we present PTMExplorer, an interactive platform for the multi-dimensional visualization of protein PTMs. PTMExplorer comprises three core modules: PTM Inspector, built upon ProtVista, provides a multi-track, sequence-feature integrated view incorporating intrinsically disordered region (IDR) prediction (via flDPnn), surface accessibility calculation (via FreeSASA), and UniProt functional annotations; PTM 3D Locator, leveraging the Nightingale/Mol* engine, anchors modification sites onto AlphaFold/Protein Data Bank (PDB) 3D structures through residue mapping via PDBe-SIFTS; and PTM Overview, utilizing the R circlize package, presents a panoramic polar circos plot illustrating modification distribution and inter-group differential regulation. Additionally, three major disease-associated modification databases (PTMD, qPTM, and PhosCancer) are integrated as PTM-Disease Nexus, enabling co-localization comparison between user-defined differential sites and reported disease-related sites. PTMExplorer currently supports eight model organisms, accepts user-uploaded differential analysis results, and provides multi-dimensional annotations and various visualization options (https://www.bioladder.cn/PTMExplorer/). Using a multi-omics dataset from hepatocellular carcinoma (18 patients, 9 modification types) as a case study, we demonstrate the practical utility of PTMExplorer in screening potential biomarkers, revealing multi-modification coordination mechanisms, and distinguishing between absolute and relative quantification patterns.

bioinformatics↗