Nethobench: Forecasting design and conditioning shape the structural realism of generated neural dynamics

Abstract

Forecasting models are increasingly used to model neural population activity, yet existing benchmarks typically evaluate them by pointwise predictive fidelity to held-out observations. In stochastic, partially observed systems, agreement with one sampled realization is distinct from preserving the statistical and dynamical structure of the process. We introduce Nethobench, a benchmark that evaluates structural agreement across five metric families: distributional, temporal--spectral, relational, geometric, and state-dynamical. We validate Nethobench using targeted perturbations and stochastic systems with known dynamics, and apply it to mouse calcium imaging, mouse Neuropixels recordings, and human intracranial recordings, using empirical references to calibrate interpretation. Nethobench detects targeted structural failures and assigns high scores to independent samples from the correct synthetic process despite trajectory mismatch. It tracks structural changes during training and under domain shift. Model comparisons reveal how forecasting formulation, architecture, training regime, and loss shape structural profiles, while OmniMouse input ablations improve all five family means when behavior is added to video in one held-out session. In Brain Treebank, analog retrieval passes more structural checks while a multistep GRU better predicts the observed realization. These results motivate reporting calibrated family-level structural metrics alongside pointwise accuracy, proper probabilistic scores, and task-specific endpoints when evaluating neural forecasts across species and recording modalities.