Selection may oppose invasion, yet favour fixation: consequences for evolutionary stability

Abstract

Models of evolution by natural selection often make the simplifying assumption that populations are infinitely large. In this infinite population limit, rare mutations that are selected against always go extinct, whereas in finite populations they can persist and even reach fixation. Nevertheless, for mutations of small phenotypic effect, it is widely believed that in sufficiently large populations, if selection opposes the invasion of rare mutants, then it also opposes their fixation. Here, we identify circumstances under which infinite-population models do or do not accurately predict evolutionary outcomes in large, finite populations. We show that there is no population size above which considering only invasion generally suffices: for any finite population size, there are situations in which selection opposes the invasion of mutations of arbitrarily small effect, but favours their fixation. This is not an unlikely limiting case; it can occur when fitness is a smooth function of the evolving trait, and when the selection process is biologically sensible. Nevertheless, there are circumstances under which opposition of invasion does imply opposition of fixation: in fact, for the n-player snowdrift game (a common model of cooperation) we identify sufficient conditions under which selection against rare mutants of small effect precludes their fixation--in sufficiently large populations--for any selection process. We also find conditions under which--no matter how large the population--the trait that fixes depends on the selection process, which is important because any particular selection process is only an approximation of reality.