Assessing the robustness of SNaQ to violations induced by high-level phylogenetic networks

Abstract

Phylogenetic networks extend the traditional tree model to capture reticulate evolutionary processes such as gene flow and hybridization. Among available inference tools, SNaQ is a widely used quartet-based method that offers a computationally efficient, statistically grounded approach to network estimation, but is limited to level-1 networks, in which reticulation cycles do not overlap. This assumption is a statistical requirement for identifiability rather than a reflection of biological reality, as many evolutionary scenarios, particularly those involving extensive or closely spaced gene flow, are expected to produce level-2 or higher networks. How SNaQ performs when this assumption is violated remains poorly understood. Here, we systematically evaluate SNaQs performance on simulated non-level-1 networks. Because existing network comparison metrics such as hardwired cluster dissimilarity are not true distances beyond level-1, we introduce complementary measures: hybrid cluster compatibility, blob compatibility, and tree-of-blobs comparison, to more directly assess structural recovery. We find that while SNaQ does not recover the exact topology of non-level-1 networks, it reliably infers the circular order of taxa and frequently recovers a tree of blobs compatible with the true network, suggesting the level-1 constraint acts as a form of regularization against overfitting. Recovery of reticulation signal is strongly tied to inheritance proportion and the user-specified maximum number of reticulations, with SNaQ behaving as a conservative estimator that favors strong, well-supported events over finer-scale or overlapping signals. These results clarify the strengths and limits of quartet-based network inference under model misspecification and offer practical guidance for applying SNaQ to complex reticulate histories.