A Framework for Clustering and Zero-Shot Labeling of Ecological Images Using Foundation Models

Abstract

The increasing automation of biodiversity monitoring is generating large volumes of ecological images requiring taxonomic identification, creating a substantial annotation bottleneck. Assigning species-level labels requires considerable time and expert knowledge, limiting scalability. We propose a framework for unsupervised clustering and zero-shot labeling of ecological images using foundation models. The framework extracts image embeddings from pretrained models and groups images into taxonomic clusters without task-specific training. Species labels are assigned through zero-shot text matching between image embeddings and prompted information representing taxa expected in the dataset. We investigate dimensionality reduction and six clustering algorithms for organizing fine-grained imagery. An uncertainty score combines clustering and labeling information to triage images into three confidence tiers, allowing experts to focus on uncertain or potentially incorrect assignments. The framework is evaluated on four bird and moth datasets, including two annotated camera-trap collections. The camera-trap datasets, per-image species-level accuracy reached 84% on DK-Moth and 83% on HUGO-Bench Aves. An interactive report presents clusters and zero-shot labels by confidence tier and prioritizes cases requiring expert review. The framework provides a training-free approach for organizing and labeling ecological image collections, reducing manual annotation while retaining expert oversight and demonstrating the potential of foundation models for scalable, semi-automated annotation.