bioRxiv2026
The breakdown of distyly, a polymorphism that promotes disassortative pollination between two floral morphs, is often assumed to increase self-fertilization, reducing the efficacy of selection and challenging long-term species persistence. We tested whether repeated losses of distyly Linum were associated with genomic signatures of selfing by 1) testing for relaxed selective pressure in homostylous relative to distylous lineages, and 2) by characterising population genomic patterns of the homostylous Linum leonii in comparison to its distylous close relative Linum perenne. We generated whole-genome sequences and target-capture data from sixteen Linum species, and additionally built a high-quality genome assembly and acquired population-level whole-genome sequencing data for L. leonii (n=20). We reconstructed plastome phylogenies, estimated selective pressure for chloroplast and nuclear genes, inferred ancestral floral morph states, and tested for signatures of selfing in homostylous lineages. Compared to theoretical expectations, results were mixed, with partial identification of relaxed selective pressure in homostyles. Population genomic analyses of L. leonii revealed a moderate selfing rate of 0.32, suggesting that loss of distyly was associated with mixed mating rather than selfing, contrary to previous results on loss of distyly. Reduced nucleotide diversity and evidence for relaxed selection efficacy in L. leonii were instead consistent with a historical bottleneck. In Linum, the genomic consequences are more heterogeneous than generally assumed, and likely depend on species-specific evo-demographic history. This study highlights the complex evolutionary dynamics associated with the breakdown of distyly and emphasizes the need for comparative population genomic studies to clarify how such transitions shape evolutionary processes. Significance statementPlant mating system variation is central to evolution as it shapes genetic diversity, adaptability and fitness. Loss of distyly, an iconic example of a complex mating system favouring cross-pollination, can drive shifts from outcrossing to selfing, with potentially severe evolutionary consequences for the long-term persistence of the species in which it occurs. Using high-quality genome assembly and omic data for multiple Linum species, we tested for relaxed selective pressure in homostylous compared to distylous lineages, and tested for a population genomic signature of selfing in homostylous Linum leonii compared to the closely related distylous Linum perenne. Contrary to theoretical expectations, evidence for relaxation of selection was mixed in Linum homostyles and L. leonii did not exhibit a genomic signature of selfing. Our study reveals multiple evolutionary pathways following the loss of distyly, and highlights how mating system transitions, together with complex demographic processes, shape plant genetic diversity and evolution.