UniMedSeg: Unified Multi-Modal Medical Image Segmentation via Uncertainty Guided Encoding and Frequency-Spatial Dual-Prompt Decoding

Abstract

Text-driven medical image segmentation demands unified models based on multi-modal learning, capable of processing diverse imaging modalities while providing spatially reliable prediction estimates for clinical practice. Nevertheless, existing vision-language model solutions suffer from poor domain generalization and exhibit limited capability for uncertainty estimation to model the cumulative unreliability of cross-modal alignment across network layers. In this paper, we propose UniMedSeg, a unified vision-language framework for multi-modal medical image segmentation built upon frozen pre-trained vision-language backbones. Our framework incorporates modality-aware adaptation, cross-layer uncertainty-guided encoding, and a frequency-spatial dual-prompt decoder, wherein the progressive uncertainty propagation mechanism of the encoder suppresses misleading cross-modal features, whereas the decoder disentangles anatomical structure and boundary details in the frequency domain under dual textual prompts. Auxiliary losses further sharpen boundary representations and calibrate pixel-wise uncertainty maps. Extensive experiments demonstrate that UniMedSeg preserves competitive in-domain segmentation performance, improves domain generalization, and yields clinically useful spatial risk estimation.