BioFuse: An Embedding Fusion Framework for Biomedical Foundation Models

Abstract

The biomedical field has witnessed a surge in pre-trained foundation mod-els excelling in specific sub-domains such as radiology and histopathology. While integrating these models promises a more comprehensive understand-ing of biomedical data, it poses challenges in model compatibility and feature fusion. We present BioFuse, a novel open-source framework designed to generate optimised biomedical embeddings. BioFuse utilises a pool of 9 state-of-the-art foundation models to create task-specific embeddings. It employs grid search to automatically identify the optimal combination of models, fusing their embeddings through vector concatenation. On the MedMNIST+ benchmark, using XGBoost as the downstream classifier, Bio-Fuse outperforms several existing methods, achieving SOTA AUC in 5/12 datasets while maintaining near-SOTA performance across most remain-ing datasets. Remarkably, our experiments reveal unexpected cross-modal capabilities, with histopathology and radiology models showing strong performance when applied to other imaging modalities. BioFuse features a high-level API1 for immediate deployment and an extensible architecture to incorporate future models and fusion techniques. We anticipate BioFuse will not only enhance the utility of foundation models in biomedicine but also open new avenues for uncovering cross-modal relationships in biomedical data.