A Sense of Direction: Detecting Natural Selection Across Non-Stationary Genomes

Abstract

Codon substitution models detect natural selection by comparing non-synonymous (dN) and synonymous (dS) rates at each site. Standard methods assume stationary nucleotide composition across the phylogeny. In many genomes, however, GC-biased gene conversion (gBGC), codon usage bias, strand asymmetry, and host deaminases drive directional shifts in base composition along lineages. Under a stationary Muse-Gaut model with site-to-site synonymous rate variation (FEL), position-specific base frequencies and site-specific dS absorb most tree-wide drift. As a result, selection calls remain unchanged at 96%-99% of codons. At the remaining boundary sites, however, directional shifts bias dS in opposite directions depending on the ancestral codon. Rapid silent turnover at codons shifting toward the favored base inflates dS, producing apparent purifying selection and increasing across-site synonymous rate variance. Conversely, when amino acid replacements oppose the compositional tide, stationary models miss diversifying selection. Branch-specific models can accommodate shifting base frequencies, but diagonalizing a separate non-reversible codon rate matrix on every branch is too slow for large trees. Here we introduce Directional Fixed Effects Likelihood (dFEL). The method first tests whether base composition varies across taxa and runs standard FEL when frequencies are uniform. When composition shifts across lineages, dFEL roots the tree by maximum likelihood without an outgroup and models each branch as a smooth weighted average of three regime-specific transition probability matrices representing AT-enriching, neutral, and GC-enriching evolution. Sharing these matrices across the tree requires only three matrix diagonalizations per site. We validate dFEL across 1,800 simulated alignments and 87 unique empirical alignments (93 benchmark analyses) spanning up to 869 vertebrate species. It maintains calibrated false-positive rates, improves power, cuts whole-gene synonymous rate variance by up to 63%, and recovers established roots from coding sequences alone (https://github.com/veg/hyphy-3).