Evaluating sequence-to-function deep learning models for ancestry-stratified regulatory variant effect prediction using multi-ancestry blood eQTLs
BackgroundSequence-to-function (S2F) deep learning models are increasingly used to prioritize non-coding regulatory variants, but their behavior across ancestrally diverse populations remains unclear. Because both training data and reference resources are heavily European-centered, multi-ancestry benchmarks are needed to determine whether S2F scores capture regulatory effects consistently across populations with different allele-frequency and LD patterns. MethodsWe evaluated Borzoi and AlphaGenome using whole blood eQTL data from the MAGENTA cohort, including African American (AA; N = 224), Caribbean Hispanic (CH; N = 209), and Non-Hispanic White (NHW; N = 235) participants. Model predictions were benchmarked against sampled nominal eQTLs and ancestry-stratified SuSiE fine-mapped variants using Spearman correlation, direction concordance, inter-model convergence, and distance-matched AUROC, with sensitivity analyses for minor allele frequency and comparison-set definition. We also compared FILER functional annotation overlap among high-Posterior Inclusion Probability (PIP) variants across ancestries. ResultsBoth models showed weak agreement with nominal eQTL effect sizes across ancestries and TSS-distance bins ({rho} [≤] 0.138), with direction concordance only marginally above chance. Agreement and discrimination improved for high-confidence fine-mapped variants, and Borzoi and AlphaGenome showed stronger inter-model convergence on fine-mapped variants than on nominal eQTLs, consistent with enrichment for regulatory variants whose effects are more apparent to sequence-based models. In distance-matched AUROC analyses at PIP [≥] 0.9 using PIP < 0.01 variants as low-PIP comparison variants, the AA high-PIP variant set yielded the highest discrimination for both Borzoi (0.837 [95% CI: 0.790-0.870]) and AlphaGenome (0.820 [0.793-0.845]). The CH-versus-NHW ordering was model-dependent: Borzoi yielded higher AUROC in NHW than CH, whereas AlphaGenome produced nearly identical CH and NHW estimates. AUROC values were lower when intermediate-PIP variants were used as comparison variants, but the AA set retained the highest discrimination. MAF-stratified sensitivity analyses attenuated some ancestry contrasts but did not eliminate the higher AA discrimination pattern. Functional annotation analysis showed that AA high-PIP variants more often overlapped chromatin accessibility and chromatin-contact annotations than NHW variants, despite lower overlap with prior eQTL and sQTL annotation catalogs. ConclusionsBorzoi and AlphaGenome showed limited agreement with nominal eQTL effect sizes, but better distinguished high-confidence fine-mapped eQTLs from low-PIP variants. These results support using S2F scores as prioritization evidence for fine-mapped regulatory variants, especially promoter-proximal high-PIP variants, rather than as standalone predictors of eQTL effect size. The strongest discrimination was observed for the AA high-PIP variant set. Overall, the AA result is best interpreted as stronger separation of high-PIP variants from lower-PIP comparison variants, shaped by fine-mapping resolution, LD, the choice of comparison variants, and annotation composition.