The Sequential Multispecies Coalescent

Abstract

The multispecies coalescent (MSC) model applies coalescent theory to gene evolution within and among reproductively isolated populations ("species") to estimate a species tree in the face of gene tree conflict resulting from deep coalescence. Sequential Monte Carlo (SMC) uses particle filtering to sample a posterior distribution, providing a fully-Bayesian and easily parallelized alternative to traditional MSC tree inference approaches. The method we propose samples first from the joint posterior distribution of gene and species trees, then samples species trees conditional on gene trees sampled previously, employing SMC for both rounds. Analyses of simulated and empirical datasets yield results comparable to state-of-the-art Bayesian MCMC approaches. Sampling the multispecies coalescent using SMC retains the advantages of fully Bayesian methods and is parallelizable in ways that Bayesian MCMC methods are not but also adds unique challenges. We demonstrate the performance of SMC compared to other commonly-used species tree methods using two empirical datasets and 400 simulated datasets.