Mutation Order and Selection Shape Intratumor Heterogeneity in Tumor Evolution

Abstract

Cancer progression often requires the accumulation of multiple driver mutations, yet the same set of drivers may be acquired in different orders. How these alternative mutation-order pathways jointly shape tumor clonal structure remains unclear. We develop a multitype branching-process model in which malignant transformation requires two driver mutations and distinguish malignant cells both by their mutation order and by the independent transformation event that founded their clone. Under a successive exponential approximation, we establish point-process limits for the pathway-specific clone-size processes and derive a closed-form expression for the limiting expected Simpson's index of the combined malignant population. When the two mutation orders produce malignant cells with the same net growth rate, the limiting index separates into effective pathway weights, determined by mutation rates and birth-death dynamics at preceding stages, and within-pathway concentration terms, determined by intermediate-to-malignant growth-rate ratios. This decomposition shows that a driver's effect on heterogeneity depends critically on when it is acquired. A strong driver acquired early expands the intermediate lineage and increases the supply of independent malignant founders, whereas the same driver acquired at the final transition strengthens the growth and age advantage of early-founded malignant clones. Under additive fitness effects, these counteracting mechanisms can produce a non-monotone relationship between selective advantage and clonal concentration. We further show that threshold-like non-additive fitness effects can generate highly concentrated malignant populations, while order-dependent terminal fitness leads the faster-growing pathway to dominate asymptotically. Together, these results reveal how mutation order, mutational accessibility, selection, and epistasis jointly determine lineage-level intratumor heterogeneity.