Declines in Neanderthal and Denisovan population sizes are probably not an artifact of subdivision

Abstract

PSMC estimates of Neanderthal effective population size (Ne) exhibit a roughly 5-fold decline across the most recent 20 ky before the death of each fossil. To explain this pattern, this article develops new theory relating genetic variation to geographic population structure and local extinction. It argues that the observed pattern results from subdivision and gene flow. If two haploid genomes are sampled from the same subpopulation, their recent ancestors are likely to be geographic neighbors and therefore coalesce rapidly. By contrast, remote ancestors are likely to be far apart, and their coalescent rate is lower. Consequently, Ne is larger in the distant past than in the recent past. New theoretical results show that modest rates of extinction cause substantial reductions in heterozygosity, Wrights FST, and Ne.