bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.64898/2026.05.28.728616

SortIT - A Tool For Assessing Observer Variability And Creating Ground Truth Image Classification Datasets

Uegami, W.·Bisson, T.·Okoshi, E. N.·Costa da Silva, F. G.·Jiragawasan, C.·Zerbe, N.·Bychkov, A.·Fukuoka, J.
View original source ↗Export metadata ↓

Abstract

Interobserver variability in pathological assessments is a well-recognized challenge that impacts diagnostic reliability and disease understanding. This variability exists across many subspecialties due to the subjective nature of evaluations. Artificial intelligence (AI) applied to whole slide images has potential to standardize procedures and reduce variability in pathology, but transitioning to these technologies does not guarantee improvement. Establishing reliable ground truth datasets with consensus annotations is crucial for developing robust AI solutions. We introduce SortIT, an open-source web application that facilitates systematic creation and evaluation of ground truth image tile annotations. SortIT enables multiple annotators to independently label tiles, with flexible user permission controls. Annotated data can be exported for statistical analysis of observer variation and for creating ground truth datasets from consensus tiles. We outline protocols using SortIT for several use cases: (1) mitosis segmentation in tumor regions, (2) evaluating AI solutions for prostate cancer grading by comparing to expert consensus, and (3) granuloma classification by annotating discriminative tile-level features. Key strengths of SortIT lies in its ease of deployment, making it accessible and usable for a wide range of users. Overall, SortIT provides a valuable tool to establish high-quality ground truth datasets and comprehensively assess observer variability. Critical evaluation of ground truth quality using systematic annotation methodologies is crucial for developing accurate and generalizable diagnostic AI tools. Its open-source nature facilitates community adoption and further development.

Explore related subjects

pathologyPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Uegami, W., Bisson, T., Okoshi, E. N., Costa da Silva, F. G., Jiragawasan, C., Zerbe, N., Bychkov, A., Fukuoka, J.. 2026-05-29. SortIT - A Tool For Assessing Observer Variability And Creating Ground Truth Image Classification Datasets. https://doi.org/10.64898/2026.05.28.728616

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Dysregulated Platelet GPIb alpha - VWF Signalling in Abdominal Aortic Aneurysm formation and Progression

Background: Platelets are critical drivers of thrombo-inflammatory responses in different cardiovascular diseases. Abdominal aortic aneurysm (AAA) is a progressive, life-threatening vascular disorder mainly characterised by chronic inflammation, extracellular matrix degradation, and the formation of a platelet-rich intraluminal thrombus (ILT). Experimental and clinical evidence identified platelets as main players in AAA pathology as evidenced by elevated platelet activation and procoagulant activity that critically contribute to AAA progression. Methods: The present study investigated the contribution of glycoprotein (GP)Ib alpha, the von Willebrand factor (VWF)-binding subunit of the platelet GPIb-IX-V complex, to AAA initiation and progression in experimental AAA using the ePPE mouse model and in patients. Results: Genetic ablation of platelet GPIb alpha significantly attenuated early aneurysm expansion in experimental AAA, indicating a critical role for GPIb alpha during the initial stages of aneurysm development. This initial effect was compensated at later time points showing no differences in aneurysm progression between groups. Notably, genetic deletion of GPIb alpha induced a constitutively hyperactive platelet phenotype already in naive mice that was further amplified during experimental AAA. This elevated platelet hyperactivity was mainly due to increased GPVI activation of platelets 28 days post-surgery. To assess the clinical relevance, spatial profiles of human ILT specimens from patients with AAA were analysed. In the ILT, we detected a highly compartmentalised distribution of GPIb alpha and VWF with pronounced enrichment within the luminal layer. In parallel, circulating VWF activity as well as platelet surface expression of GPIb alpha were significantly increased in patients with AAA. Conclusion: Collectively, these findings identify a dysregulated GPIb alpha-VWF axis in human AAA pathology, mainly characterised by enhanced platelet GPIb alpha surface expression and increased activity of circulating VWF.

pathology↗
bioRxiv2026

OmiCoreTumorDetector: an open, molecularly validated model for mapping tumour regions in colorectal cancer H&E sections

Defining tumour regions on haematoxylin and eosin (H&E) sections is a routine first step in spatial-omics studies, yet it is usually done by hand and is difficult to reproduce. We present OmiCoreTumorDetector, an openly licensed model that maps tumour-enriched regions in colorectal cancer (CRC) H&E sections and exports them as QuPath-compatible annotations. The released model (omicore-tumordetector-crc-he-v0.1) is an ensemble of three convolutional classifiers trained on 100,000 public tissue tiles, combined with Macenko stain normalisation at inference. During development we found that the main obstacle to reuse was calibration under stain-domain shift rather than discrimination: a single model kept an area under the ROC curve (AUROC) of 0.955 on unseen slides while its sensitivity at the conventional 0.5 threshold fell to 0.48. Training on non-normalised tiles raised tumour AUROC on an independently collected tile set from 0.836 to 0.992, and normalising at inference reduced false-positive tumour area in normal-adjacent tissue by 16- to 26-fold. On five 10x Visium HD CRC sections that share no material with the training data, the released model called 27.7-48.5% of tissue as tumour in three carcinoma sections and 0.08% and 1.82% in two normal-adjacent sections, exporting no tumour region from either normal section. On the carcinoma section with matched single-cell-resolution transcriptomics, agreement with transcriptome-derived tumour-cell identities reached an AUROC of 0.985 (95% spatial-block bootstrap CI 0.975-0.993). The image model never observes gene expression, so this is orthogonal evidence. The model localises tumour-enriched regions at 112 um resolution; it does not identify individual malignant cells and has not yet been validated across scanners, institutions or histological variants. Code, weights and evaluation are released under Apache-2.0 and installable with pip install omicoretumordetector.

pathology↗
bioRxiv2026

Thyroid Dysfunction in Male Patients at Asia Med Laboratory, Herat, Afghanistan July 2021-Jan 2022

Objective: Hyperthyroidism and hypothyroidism related to iodine deficiency are major public health concerns in Afghanistan. This study aimed to assess the frequency of thyroid dysfunction among male patients referred for thyroid testing and its association with age, and to examine monthly trends in thyroid dysfunction at Asia Med Laboratory in Herat, Afghanistan, from July 2021 to January 2022. Methods: A retrospective analysis was conducted on 250 male patients aged 0-69 years. We measured Serum TSH, total T4, and total T3 levels, and thyroid status was classified using age specific reference ranges. In addition, the frequency of thyroid dysfunction was analyzed across age groups with monthly trends of thyroid state. Results: Overall, the euthyroid state consisted of 69.2% of participants, 24.8% with overt hypothyroidism, 3.2% with overt hyperthyroidism, and 2.8% with subclinical hyperthyroidism. Thyroid status differed significantly by age (p = 0.0135), with hypothyroidism increasing in older age groups and reaching its highest proportion among men aged 60-69 years (55.6%). Euthyroidism predominated in patients aged 10-39 years, while hyperthyroidism across age groups remained relatively infrequent. After September 2021, a threefold increase was observed in the total number of male patients referred for thyroid testing. During this period, the proportion of hyperthyroidism increased slightly, whereas hypothyroidism cases declined. Conclusion: In conclusion, hypothyroidism was more frequent with older age. The rise in absolute case numbers after September 2021 likely reflects increased patient referrals, underscoring the need for ongoing monitoring of thyroid function. The study may assist in the early management of thyroid disorders and in reducing their complications.

pathology↗