Haplotype-based models improve sweep detection in ancient populations with complex demography

Abstract

Identifying signatures of positive selection in humans is complicated by demographic processes such as bottlenecks, migration and admixture, all of which can distort or obscure the genomic patterns produced by selective sweeps. Ancient DNA offers a direct window into past allele and haplotype frequencies, yet most sweep scans in ancient populations rely on allele-frequency or site frequency spectrum (SFS) summaries, with limited use of haplotype-based approaches. Here, we evaluate the performance of haplotype and SFS-based methods for detecting selective sweeps under demographic scenarios that reflect the complex history of ancient and modern Europeans. We extend the haplotype-based likelihood framework saltiLASSI to accommodate pseudohaploid ancient genomes, enabling the use of truncated haplotype frequency spectra and their spatial decay to detect sweeps without requiring phased data. Using forward-in-time simulations, we examine sweeps of varying ages, two pulses of admixture with different source proportions, and cases where selection continues or ceases after admixture. We compare saltiLASSI to a widely used SFS-based approach (SweepFinder2). Our results show that haplotype-based likelihood models retain higher power than SFS methods in admixed populations, particularly when sweep haplotypes are introduced through migration or when selection has not had sufficient time to regenerate a clear SFS signature after admixture. These findings highlight the promise of haplotype-based inference for ancient DNA and demonstrate how model-based approaches can improve the detection of historical selective sweeps in populations with complex demographic histories.