Gadiyar, I. | bioRxiv Science

bioRxiv2026

Contributions of Site Heterogeneity in Multi-site Brain-Phenotype Predictions

Machine learning studies of brain-phenotype associations typically summarize accuracy with a single Pearson correlation, yet in harmonized multi-site data this metric is shaped by training-set size, feature and outcome variability, and true effect heterogeneity. We analyzed structural MRI data from the Reproducible Brain Charts consortium. After preprocessing and quality control, 3486 participants from 9 sites were retained. Three training set compositions were used to understand contributions of site heterogeneity after harmonization: within-site (Local), size-downsampled multi-site (Multi-Down), and full multi-site (Multi-Full). Another site-agnostic strategy (Overall) pooled all sites into a single dataset. All strategies included the target site in the training and testing samples. These analyses separate the contribution of the site effect from that of sample size. Between-site heterogeneity was extensive (age I^2 {approx} 95%). The overall correlation of age prediction (r = 0.73) exceeded the within-site meta-analytic estimate (Multi-Full r = 0.63). The improvement of correlation in the pooled dataset was driven by a mechanical inflation from the larger outcome variance. At matched sample size, within-site training outperformed harmonized multi-site training. Increasing the multi-site training sample size did not yield a significant advantage over within-site training. The apparent advantage of cross-dataset training thus appears to reflect training-set size rather than a benefit of multi-site composition per se. The same pattern generally held for the weaker brain-p-factor signals. Together, these results indicate that between-site heterogeneity in machine learning brain-phenotype prediction persists after harmonization. We recommend reporting site-level prediction accuracy and feature dispersion and standard deviation of the outcome variable alongside any pooled estimate to improve inter-site and inter-study comparison of machine learning prediction accuracy.

neuroscience↗

bioRxiv2025

Improved Estimation of Correlation Accuracy for Machine Learning Brain-Phenotype Associations

Machine learning is used in neuroscience to examine brain-phenotype associations and facilitate individual prediction from high-dimensional brain imaging. For continuous phenotypes, Pearsons correlation between the observed and predicted phenotype is used to quantify model accuracy in testing data. However, recent research suggests millions of samples may be needed to reliably estimate the maximum achievable predictive accuracy (MAPA). We formally define the MAPA and show that Pearsons estimator is biased for this quantity and its confidence intervals fail to capture the target. We develop a semiparametric (double machine learning) one-step estimator that more accurately estimates the MAPA and yields valid confidence intervals across flexible machine learning settings. Analyzing data from the Reproducible Brain Charts dataset, we show that this estimator has smaller bias when estimating brain-phenotype associations of neuroimaging data with age and psychopathology phenotypes. We show that MAPA for psychopathology factor scores using machine learning models built on structural and functional imaging measures is not better than using demographic and nuisance covariates alone.

neuroscience↗