Benchmarking the AlphaGenome Variant Impact score against SpliceAI on post-release ClinVar classifications and experimental splicing assays

Abstract

The AlphaGenome Variant Impact (AVI) score ranks variants by predicted impact, and its developers report large gains over other genome-wide scores for intronic ClinVar variants. We tested AVI on two kinds of ground truth the developers did not use, under an analysis plan set before scoring: ClinVar classifications made after the developers' ClinVar release (87,666 rare single-nucleotide variants outside AVI's training chromosomes), and six splicing assays whose variants no predictor chose. In intronic variants at least 3 bases from an exon (449 pathogenic or likely pathogenic, 28,188 benign or likely benign), AVI reached an area under the receiver operating characteristic curve (AUROC) of 0.957. It did not differ detectably from SpliceAI (difference -0.005, 95% interval -0.017 to 0.006) or Pangolin (-0.006, -0.018 to 0.007); both intervals exclude differences larger than about 0.02. AVI had a higher AUROC than CADD (+0.071). In deep intronic variants it trailed SpliceAI (-0.070, -0.118 to -0.026). Under an adapted ClinGen calibration, AVI gave strong evidence to 52% of pathogenic variants on the pooled test set, but its levels did not hold in every region. It gave benign evidence to 24% of benign variants, against 90% for SpliceAI. On the assays, AVI ranked splice-altering variants below raw SpliceAI (pooled difference -0.089, -0.129 to -0.049) and did not differ from masked SpliceAI (+0.003). SpliceAI2, released during the study, led AVI on both kinds of ground truth. AVI did not outperform splicing predictors on AUROC and gave little benign evidence. AlphaGenome outputs are provided for research use, not for diagnostic decisions.