A Reproducible and Extensible Benchmark of Supervised Cell Type Annotation Tools for Cytometry Data

Abstract

High-dimensional cytometry technologies such as flow cytometry (FCM) and mass cytometry (CyTOF) are central to immunophenotyping in research and clinical practice. While manual gating remains the standard for cell population annotation, it is time-consuming, difficult to scale, and subject to inter-operator variability. Supervised annotation methods have emerged as a way of scaling manual annotation work, yet independent benchmarks for comparing these tools remain limited and quickly become outdated. This study presents a reproducible and extensible benchmark of supervised cytometry annotation tools implemented within the OmniBenchmark framework. Five supervised annotation methods were evaluated, spanning linear models, nearest-neighbor approaches, tree-based classifiers, mixture-rule systems, and deep learning, across eight publicly available datasets carefully selected to cover technologies, tissues, panel designs, and healthy and disease contexts. Using a sample-centric cross-validation design that reflects common reference-mapping scenarios, overall and per-population F1 scores, performance on rare populations, runtime, and robustness to reduced training set sizes was tested. Performance varied substantially across datasets and was not fully explained by dataset size or dimensionality, highlighting both operator dependence in annotation and the importance of biological context, cohort heterogeneity, and population imbalance. Less prevalent populations (<1%) remained a key challenge for most methods. Downsampling analyses showed that moderate reference sizes were often sufficient to achieve near-maximum performance. Rather than ranking methods, this benchmark provides a standardized and transparent framework for evaluating annotation tools under realistic deployment conditions. As a living resource, the OmniBenchmark implementation supports continuous integration of new datasets, tools, and metrics for both tool developers and end users annotating datasets. This enables ongoing, reproducible method comparison and informed tool selection for diverse cytometry applications.