The IICR and the Non-Stationary Structured Coalescent: demographic inference with arbitrary changes in population structure

Abstract

In the last years, a wide range of methods allowing to reconstruct past population size changes from genome-wide data have been developed. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those produced under models of population size change. Recently, Mazet et al. (2016) showed that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size changes, having exactly the same distribution of T2 (the coalescence time for a sample of size two) to that of the structured model. They called this function IICR (Inverse Instantaneous Coalescence Rate) and showed that it does not necessarily correspond to population size changes under non panmictic models. Besides, most of the methods used to analyse data under models of population structure tend to arbitrarily fix that structure and to minimise or neglect population size changes. Here we extend the seminal work of Herbots (1994) on the structured coalescent and propose a new framework, the Non-Stationary Structured Coalescent (NSSC) that incorporates demographic events (changes in gene flow and/or deme sizes) to models of nearly any complexity. We show how to compute the IICR under a wide family of stationary and non-stationary models. As an example we address the question of human and Neanderthal evolution and discuss how the NSSC framework allows to interpret genomic data under this new perspective.\n\nAuthor summaryGenomic data are becoming available for a rapidly increasing number of species, and contain information about their recent evolutionary history. If we wish to understand how they expanded, contracted or admixed as a consequence of recent and ancient environmental changes, we need to develop general inferential methods. Currently, demographic inference is either done assuming that a species is a single panmictic population or using arbitrary structured models. We use the concept of IICR (Inverse of the Instantaneous Coalescence Rate) together with Markov chains theory to develop a general inferential framework which we call the Non-Stationary Structured Coalescent and apply it to explain human and Neanderthal genomic data in a single structured model.