bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.64898/2026.08.04.742766

The influence of incompatibilities and heterosis on hybrid population genetics

Ayala-Lopez, J. A.·Peischl, S.·Bank, C.
View original source ↗Export metadata ↓

Abstract

A long-standing question in evolutionary biology is: Under what circumstances can speciation occur despite hybridisation or because of hybridisation? Some models of hybrid incompatibilities predict that speciation can occur even in the presence of gene flow and strong selection against hybrids, an outcome also influenced by genetic contributions from parental species and genetic architecture. On the other hand, empirical work has shown that heterosis can counteract the effect of incompatibilities, hindering the speciation process. Theoretical models that simultaneously consider the positive and negative impacts of hybridisation on fitness remain scarce, raising questions about the effects of hybrid incompatibilities in the presence of heterosis. To address this question, we study how (Bateson)-Dobzhansky-Muller incompatibilities (BDMIs) interact with overdominant mutations in a two-locus population genetics model of an isolated hybrid population. We find that the strength of overdominance relative to incompatibilities determines the long-term genetic composition of the hybrid population. We show that high recombination exposes incompatibilities to selection and reduces the frequency of derived alleles in the hybrid population, limiting the strength of BDMIs that can be maintained by the balance with overdominance. We also show how neutral variation is affected by the strength of selection and the recombination rate between incompatible loci, generating patterns that include an increase in local variation resembling associative overdominance, or a reduction resembling background selection. Such variation of neutral variation, particularly at intermediate distances from BDMI loci, can generate peaks or troughs of diversity that are explained by the recombination rate between BDMI loci, and initial proportions of admixture between parental populations. Our work demonstrates how the genomic conflict caused by the interplay of overdominance and hybrid incompatibilities, recombination, and parental contributions, shape the genome of an isolated hybrid population.

Explore related subjects

evolutionary biologyPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ayala-Lopez, J. A., Peischl, S., Bank, C.. 2026-08-09. The influence of incompatibilities and heterosis on hybrid population genetics. https://doi.org/10.64898/2026.08.04.742766

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

RELAX does not reproduce its own estimates at default settings, and its output does not show it

Selection-intensity estimates from RELAX are reported as a point value of K with a likelihood-ratio P. We report that, at default settings and on data of ordinary size, the program does not reproduce its own fits. Of 27 enzyme entries refitted under two optimiser configurations, none reproduced its log-likelihood to within 0.01 units; the median change was 103 units, the largest over 3,400, and four verdicts reversed. Eighty null orthologues reproduced none. A byte-identical command returned a distinct likelihood on every repetition, single-threaded, across three releases, and on alignments simulated under the fitted model, where 3.3 per cent of replicates reproduced. The documented random-number seed never reaches the generator when assigned on the command line, yet reads back as the value supplied. PAML localises the cause: its two-ratio model, without site classes, reproduced its log-likelihood for all 288 genes; its site-class models agreed for 27 to 67 per cent. The instability follows the mixture over sites, not the program. The output does not show it: 46 of 410 fits ended with a negative likelihood-ratio statistic, impossible under convergence, and 123 of 410 report a K re-estimated under a domain restriction rather than the unconstrained maximum. Of 234 published studies using RELAX, none reported a seed. Seeding while holding the thread count at one reproduced sixty of sixty runs on twenty genes under two releases; the seed alone reproduced none of five, and no documentation states the second condition. We recommend that fits be repeated and their dispersion published.

evolutionary biology↗
bioRxiv2026

Sequential accumulation of adaptive alleles forms an inversion supergene in deer mice

Supergenes are clusters of co-inherited loci that affect multiple or complex phenotypes. Despite the growing number of chromosomal inversions identified as supergenes in natural populations, their molecular basis and evolutionary history often remain obscure. Here, we identified two candidate genes, Slc45a2 and Npr3, within a 41-Mb inversion supergene in the deer mouse (Peromyscus maniculatus) that respectively drive darker coats and longer tails - two traits associated with forest adaptation. Mice homozygous for the inversion (inv/inv) exhibit elevated Slc45a2 expression in melanocytes relative to the congenic standard genotype (std/std), disrupting pheomelanin production. In parallel, downregulation of Npr3 in inv/inv mouse growth plates prolongs postnatal growth of caudal vertebrae, resulting in tail elongation. Population-level analyses further implicate that this supergene arose through the subsequent accumulation of the Npr3 allele within the inversion, rather than by capturing all beneficial mutations at its origin.

evolutionary biology↗
bioRxiv2026

Toxin structure shapes palatability in a chemically defended butterfly

The toxicity of chemical defences is well studied, but the potential contribution of compound structure to predator deterrence remains largely unexplored. Whether predation acts more strongly on toxicity or unpalatability remains largely untested, partly because few systems allow toxin structure to vary independently of quantity. Heliconius sara larvae provide such a system: those reared on Passiflora auriculata sequester cyclopentenyl cyanogenic glucosides (CGs), while those reared on P. biflora biosynthesise comparable quantities of aliphatic CGs. Using two invertebrate predators, Camponotus floridanus ants and Hierodula membranacea mantids, we tested whether this structural difference affects palatability independent of toxicity. Mantids rejected larvae with cyclopentenyl CGs more often than larvae with aliphatic CGs, despite no detectable difference in total CG content. This pattern was mirrored in extract-based assays with ants, independently of cyanide release: extracts with cyclopentenyl CGs remained deterrent, while extracts with aliphatic CGs did not differ in deterrence from water. Live larvae, by contrast, elicited similar responses from ants regardless of CG structure. These results show that variation in toxin structure can strongly affect palatability, with some compounds conferring greater protection than others. This demonstrates the importance of chemical structural diversity in the evolution of chemical defences.

evolutionary biology↗