Reciprocal sign epistasis and sign epistasis may not be found in multi-peaked landscapes

Abstract

Fitness landscapes provide a rigorous mathematical framework for analyzing evolutionary dynamics, including the study of epistasis, the main obstacle to predicting phenotype from genotype. In 2011, Poelwijk et al. formulated a foundational theorem stating that in any multi-peaked fitness landscape, "at least two mutations exhibit reciprocal sign epistasis" (Poelwijk et al., J. Theor. Biol., 272:141). The proof relied on the implicit assumption that neutral mutations are absent, commonly accepted in theoretical studies in evolutionary biology. In this study, we extend Poelwijk et al.s analysis by incorporating genotypes with equal fitness, specifically, accounting for neutral mutations. We demonstrate that when neutral mutations are considered, conventional pairwise reciprocal sign epistasis (RSE) may be entirely absent from a multi-peaked landscape. Instead, RSE is guaranteed only when considering "distant" RSE defined through composite mutations, wherein groups of mutations are treated collectively across all their possible combinations. Applying these concepts to empirical fitness landscapes faces a practical limitation: phenotypic measurements contain experimental noise, making mutational effects statistically indistinguishable from zero. Under such conditions, statistically significant detection of RSE in multi-peaked landscapes may be impossible even when composite mutations are considered. Theoretically, our findings imply that in the presence of neutral mutations, compensatory mutations in a multi-peaked fitness landscape need not be adjacent; rather, compensation can occur following one or more neutral steps along an evolutionary path. Practically, in real-world scenarios where fitness measurements contain uncertainty, there may be a fundamental technical limitation to detecting RSE in a statistically significant manner within multi-peaked landscapes.