bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2022.05.05.487569

Systematic benchmarking of all-in-one microbial SNP calling pipelines

Falconer, C.·Cuddihy, T.·Beatson, S.·Paterson, D.·Harris, P.·Forde, B.
View original source ↗Export metadata ↓

Abstract

Clinical and public health microbiology is increasingly utilising whole genome sequencing (WGS) technology and this has lead to the development of a myriad of analysis tools and bioinformatics pipelines. Single nucleotide polymorphism (SNP) analysis is an approach used for strain characterisation and determining isolate relatedness. However, in order to ensure the development of robust methodologies suitable for clinical application of this technology, accurate, reproducible, traceable and benchmarked analysis pipelines are necessary. To date, the approach to benchmarking of these has been largely ad-hoc with new pipelines benchmarked on their own datasets with limited comparisons to previously published pipelines. In this study, Snpdragon, a fast and accurate SNP calling pipeline is introduced. Written in Nextflow, Snpdragon is capable of handling small to very large and incrementally growing datasets. Snpdragon is benchmarked using previously published datasets against six other all-in-one microbial SNP calling pipelines, Lyveset, Lyveset2, Snippy, SPANDx, BactSNP and Nesoni. The effect of dataset choice on performance measures is demonstrated to highlight some of the issues associated with the current available benchmarking approaches. The establishment of an agreed upon gold-standard benchmarking process for microbial variant analysis is becoming increasingly important to aid in its robust application, improve transparency of pipeline performance under different settings and direct future improvements and development. Snpdragon is available at https://github.com/FordeGenomics/SNPdragon. Impact statementWhole-genome sequencing has become increasingly popular in infectious disease diagnostics and surveillance. The resolution provided by single nucleotide polymorphism (SNP) analyses provides the highest level of insight into strain characteristics and relatedness. Numerous approaches to SNP analysis have been developed but with no established gold-standard benchmarking approach, choice of bioinformatics pipeline tends to come down to laboratory or researcher preference. To support the clinical application of this technology, accurate, transparent, auditable, reproducible and benchmarked pipelines are necessary. Therefore, Snpdragon has been developed in Nextflow to allow transparency, auditability and reproducibility and has been benchmarked against six other all-in-one pipelines using a number of previously published benchmarking datasets. The variability of performance measures across different datasets is shown and illustrates the need for a robust, fair and uniform approach to benchmarking. Data SummaryO_LIPreviously sequenced reads for Escherichia coli O25b:H4-ST131 strain EC958 are available in BioProject PRJNA362676. BioSample accession numbers for the three benchmarking isolates are: O_LIEC958: SAMN06245884 C_LIO_LIMS6573: SAMN06245879 C_LIO_LIMS6574: SAMN06245880 C_LI C_LIO_LIAccession numbers for reference genomes against the E. coli O25b:H4-ST131 strain EC958 benchmark are detailed in table 2. C_LIO_LISimulated benchmarking data previously described by Yoshimura et al. is available at http://platanus.bio.titech.ac.jp/bactsnp (1). C_LIO_LISimulated datasets previously described by Bush et al. is available at http://dx.doi.org/10.5287/bodleian:AmNXrjYN8 (2). C_LIO_LIReal sequencing benchmarking datasets previously described by Bush et al. are available at http://dx.doi.org/10.5287/bodleian:nrmv8k5r8 (2). C_LI O_TBL View this table: org.highwire.dtl.DTLVardef@f2446corg.highwire.dtl.DTLVardef@16a299corg.highwire.dtl.DTLVardef@d1c7a5org.highwire.dtl.DTLVardef@8a3cbaorg.highwire.dtl.DTLVardef@198f455_HPS_FORMAT_FIGEXP M_TBL O_FLOATNOTable 2.C_FLOATNO O_TABLECAPTIONReference genomes used in the EC958 benchmarking dataset and the percent identity against the three included samples. C_TABLECAPTION C_TBL

Explore related subjects

bioinformaticsPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Falconer, C., Cuddihy, T., Beatson, S., Paterson, D., Harris, P., Forde, B.. 2022-05-06. Systematic benchmarking of all-in-one microbial SNP calling pipelines. https://doi.org/10.1101/2022.05.05.487569

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Uncovering Viral Signals Across Cell Populations in Colorectal Cancer

Viral sequences have been detected in colorectal cancer, but their cellular and anatomical distribution remains poorly resolved. We aimed to establish a cell-type-resolved map of viral reference-associated RNA signals across colorectal cancer, tumour-adjacent normal tissue, and healthy colon. We repurposed human-unmapped reads from single-cell RNA sequencing libraries generated in ten publicly available studies. We retained cell-barcode and unique molecular identifier information, removed duplicate molecules, and aligned reads to a curated human-associated viral reference using three aligners. Viral signals were analysed across cell types, tissue conditions, and colorectal segments. We used participant-grouped repeated cross-validation to evaluate tissue-condition classification and screened an independent paired colorectal cancer proteomics cohort for compatible peptide matches. The study included 547 libraries from 112 participants, including 49 with matched colorectal cancer and tumour-adjacent normal samples, and approximately 415,000 cells spanning ten major cell populations. Viral reference-associated RNA signals differed across epithelial, immune, stromal, and vascular compartments, tissue conditions, and anatomical regions of the colon. Of 318 independent healthy-reference viral-load comparisons, seven involving six virus-cell-type groups remained significant after Benjamini-Hochberg correction, including plasma-cell HHV-7 and HHV-6B and mast-cell HCV-2. Cell-type-resolved viral features distinguished colorectal cancer, tumour-adjacent normal tissue, and healthy colon in participant-grouped validation; random forest achieved a macro-AUC of 0.957 (SD 0.028), with HERV-K113 as the leading quantitative feature. An independent proteomic screen identified peptide matches compatible with 20 candidate viral groups. We established a cell-type-resolved map of viral reference-associated RNA signals across colorectal tissue conditions and anatomical regions. By identifying candidate viral signals in epithelial and immune compartments, this map provides a basis for investigating virus-host pathways involved in tumour biology, immune regulation, and antigen-presentation states, and could help prioritise future combination strategies pairing virus-directed or virus-mimetic approaches with standard treatment or immune checkpoint blockade.

bioinformatics↗
bioRxiv2026

Statistical significance testing for spatial domain detection in spatial transcriptomics

In spatial transcriptomics, domain-detection methods group platform-specific spatial measurement units--cells, bins or multicellular spots--into spatial domains for comparing molecular programs, cellular composition and interactions among tissue regions. However, the number of candidate domains detected in the same tissue can vary with user-specified parameters or model-selection criteria. Whether candidate domains should be merged can be framed as a test of a single-domain null hypothesis, under which each gene has constant mean expression across all spatial units in the combined region. However, testing candidate domains on the same gene-expression data used to detect them may induce post-selection bias and yield anti-conservative p-values, increasing the risk of oversegmentation. To address these challenges, we introduce SigDoTS (statistical Significance testing for Domain deTection in Spatial transcriptomics), a hypothesis-testing framework for spatial domain detection. SigDoTS uses hierarchical testing to determine which candidate domains to retain or merge, with adjustment for family-wise error control. It accounts for gene-gene covariance and spatial dependence and computes p-values by rerunning the upstream domain-detection pipeline on datasets simulated under the fitted single-domain null. In simulations, SigDoTS controlled the type I error rate at the root node under the single-domain null and improved recovery of the true spatial domains when multiple domains were present. In human dorsolateral prefrontal cortex, SigDoTS retained superficial and deep Layer 6 domains but merged Layer 3 candidate domains across different initial numbers of candidate domains. In ovarian and colorectal cancers, SigDoTS retained epithelial domains with different gene-expression programs and local immune environments, together with immune, stromal and vascular domains.

bioinformatics↗
bioRxiv2026

ASD Neuro Agentic (ANA): An Agentic Framework for Feature Discovery and Cross Site Generalization in Autism fMRI Classification

Classifiers of autism spectrum disorder (ASD) from resting state functional MRI rarely generalize across sites, and many reported accuracies are inflated by data leakage.We show that an agentic artificial intelligence framework, ASD Neuro Agentic (ANA), whose language model agents propose features from the literature and researcher input while fixed tools perform every calculation, can search auditably for site independent classifiers while controlling the leakage. In 1,028 participants from 17 ABIDE I centers, ANA tested 71 pipelines and selected a ComBat harmonized ensemble of tangent space connectivity models from six parcellations and 98 public connectivity derivatives. In leave one center out testing, the ensemble reached an area under the curve of 0.814 (95% CI, 0.773 to 0.841), outperforming ASD DiagNet and ASD SAENet. Predictive connectivity was mainly reduced in ASD. Agentic AI offers a systematic and efficient approach to exploring, evaluating, and identifying pipelines as model selection grows complex.

bioinformatics↗