The Joint Impact of Deleterious Mutations, Dominance, and Gene Flow on Linked Neutral Variation in Structured Populations

Abstract

Most species are geographically structured, leaving characteristic signatures in neutral regions of the genome. These signatures can be distorted when neutral regions are linked to deleterious mutations. In such regions, purifying selection can reduce genetic diversity through Background Selection (BGS) or, for recessive mutations, increase diversity through Associative Overdominance (AOD). While the effect of BGS and AOD are well characterized in panmictic populations, their effects remain largely unexplored in structured populations. Here, we investigated an Isolation with Migration model using forward simulations across a range of migration, selection, dominance, and recombination parameters. We first used a genotype-based approach to quantify the effects of deleterious mutations on standard summary statistics ({pi}, dxy, FST, DAFi). We then showed that an Ancestral Recombination Graph-based (ARG) approach, tracking tree sequences from a sample of one diploid per deme, recovers the same patterns while directly relating genetic variation to the underlying coalescent processes. When recombination is sufficiently low, we found a BGS-driven regime for weakly codominant mutations, characterized by lower diversity and increased genetic differentiation (FST). For recessive mutations, we first identified an AOD-driven regime, characterized by increased diversity and lower FST values followed by a transition to a subsequent BGS-driven regime. Genealogies were similarly impacted by deleterious mutations: BGS shrunk coalescent times and produced a shift towards lineage sorting topologies, while AOD stretched coalescent times and produced a shift toward incomplete lineage-sorting topologies. These patterns were weakened by gene flow, with FST and topologies remaining close to expected under neutrality, while diversity and coalescence times remained robust to demography. Our results provide clear evidence of BGS, AOD, and of their transition in a structured model with gene flow. Importantly, these processes leave distinct and interpretable signatures on gene trees, highlighting the potential of ARG-based approaches for inferring linked selection and dominance in structured populations. Author summaryCharacterizing how demography and selection jointly shape genomic variation is a central question in population genetics. As deleterious mutations reduce fitness, they are continuously removed from populations by purifying selection. Through linkage, this affects nearby regions of the genome, leaving signatures of selection on linked neutral genetic diversity. While these effects are well understood in random mating populations, much less is known in structured populations. Specifically, the occurrence of Background Selection (BGS), which reduces diversity, and Associative Overdominance (AOD), which increases diversity, remains underexplored. Here, we used simulations to investigate how deleterious mutations shape genomic variation in a structured two-population isolation with migration model. By combining standard population genetic analyses with a genealogical approach based on Ancestral Recombination Graphs (ARGs), we showed that BGS and AOD leave distinct and interpretable signatures on common summary statistics and the underlying genealogies. We identified clear signatures of BGS and AOD when recombination was low and revealed a transition from AOD to BGS for recessive mutations, as the strength of selection increased. Our results highlight the importance of jointly considering demography and linked selection when interpreting genomic data and demonstrate the potential of ARGs to jointly infer demography, selection, and dominance from genomic data.