mixKIN: An ancestry-aware kinship estimation method for low coverage genomes
Genetic kinship analysis provides insights into the social and biological organization of past populations. However, kinship estimation from low-coverage genomes, which are common in ancient DNA (aDNA) studies, remains challenging, particularly in genetically heterogeneous or admixed populations. Approaches that assume a homogeneous background relatedness can bias estimates when individuals have distinct ancestry. We developed mixKIN, an ancestry-aware kinship estimation from low-coverage genomes. Rather than using a single mismatch baseline for all pairs, mixKIN estimates pair-specific baseline by weighting within-population diversity and between-population divergence according to ancestry proportions of both individuals in each pair. We compared mixKIN with correctKin and RelateAdmix, which account for population structure and admixture, and with READv2 representing a widely used approach that assumes a homogeneous population background. To benchmark the methods, we ran extensive pedigree simulations across diverse demographic scenarios, including distant-population admixture, intra-population heterogeneity, and genetic outliers, at coverages from 1x to 0.03x. mixKIN achieved higher average classification performance in admixed and heterogeneous scenarios, while maintaining comparable performance in homogeneous populations, with relationships evaluated up to third-degree. In an application to an empirical data from a medieval English cemetery, mixKIN recovered relationships involving an individual with West African and European ancestries without the need for a manual adjustment to account for admixture. These findings support ancestry-aware mismatch normalization as a practical approach to improving kinship estimation and retaining genetically diverse individuals in kinship studies.