Biology subjects

Steinmetz, S. T.

Publications and source records attributed to Steinmetz, S. T..

4 recordsLinked to original sources

bioRxiv2026

For MSTd, Autoencoding is all you need

While goal-driven artificial neural networks (ANNs) have successfully modeled important aspects of the primate ventral stream, their efficacy for the dorsal stream remains unclear. Here, we investigated how computational objectives and architectural constraints influence the neural alignment to MSTd, a dorsal area that demonstrates selectivity to complex optic flow patterns and is linked to self-motion perception. We systematically evaluated the neural alignment between 54 ANNs and Non-negative Matrix Factorization (NNMF) against key neurophysiological optic flow tuning properties of MSTd. We optimized these models on either a supervised self-motion estimation task (accuracy-optimized) or an unsupervised input reconstruction task (autoencoding) using both raw optic flow and model MT-encoded signals. Interestingly, accuracy on the self-motion task does not predict neural alignment. Instead, model performance bifurcates based on both objective and input encoding: autoencoders utilizing MT-like input signals consistently achieve superior correspondence with MSTd tuning preferences. Explicitly enforcing sparsity or non-negativity does not improve alignment; rather, these constraints often degrade the match to biological data. Furthermore, we demonstrate that neural alignment remains largely unaffected even when the pressure to generate an efficient code with few units is eased, suggesting that dimensionality reduction may not be a primary driver of MSTd-like tuning. Taken together, our results indicate that the tuning properties of MSTd are better explained by an unsupervised reconstruction-based objective than by supervised task optimization, suggesting a fundamental difference in the computational principles that govern the dorsal and ventral streams. Significance StatementGoal-driven neural networks have revolutionized our understanding of the ventral visual stream, yet their effectiveness in modeling the dorsal stream remains less clear. We systematically evaluated 54 neural network models to identify the computational principles that drive neural-like optic flow tuning in dorsal stream area MSTd. Surprisingly, we find that accuracy-optimized models fail to replicate biological tuning. Instead, models that reconstruct motion inputs from a biologically plausible MT-like representation achieve the highest consistency with MSTd neurons. These findings suggest that the organizational principles of dorsal stream area MSTd may be better explained by an unsupervised, reconstruction-based objective rather than one focused on the accuracy of self-motion estimation, suggesting a fundamental difference in the computational objectives of the two visual streams.

neuroscience↗

bioRxiv2024

Convolutional neural networks as a model for brain area MSTd

Accuracy-optimized convolutional neural networks (CNNs) have emerged as highly effective models at predicting neural responses in brain areas along the primate ventral stream, but it is largely unknown whether they effectively model neurons in the complementary primate dorsal stream. We explored how well CNNs model the optic flow tuning properties of neurons in dorsal area MSTd and we compared our results with the Non-Negative Matrix Factorization (NNMF) model proposed by Beyeler, Dutt, & Krichmar (2016), which successfully models many tuning properties of MSTd neurons. To better understand the role of computational properties in the NNMF model that give rise to MSTd-like optic flow tuning, we created additional CNN model variants that implement key NNMF constraints -- non-negative weights and sparse coding of optic flow. While the CNNs and NNMF models both accurately estimate the observers self-motion from purely translational or rotational optic flow, NNMF and the CNNs with nonnegative weights yield substantially less accurate estimates than the other CNNs when tested on more complex optic flow that combines observer translation and rotation. Despite their poor accuracy, however, neurons in the networks with the nonnegativity constraint give rise to tuning properties that align more closely with those observed in primate MSTd. Interestingly, the addition of the sparsity constraint has a negligible effect on the accuracy of self-motion estimates and model tuning properties. Across all models, we consistently observe the 90-degree offset in the preferred translation and rotation directions found in MSTd neurons, which suggests that this property could emerge through a range of potential computational mechanisms. This work offers a step towards a deeper understanding of the computational properties and constraints that describe optic flow tuning primate area MSTd. Significance StatementOne of the most exciting developments in visual neuroscience over the past decade is that convolutional artificial neural networks optimized to accurately categorize natural images effectively model neural activity in ventral visual areas of the primate brain. We explored whether accuracy-optimized neural networks account for well-established properties of MSTd, a brain area in the complementary primate dorsal stream that is involved in self-motion perception during navigation. Our findings indicate that such networks depart substantially from MSTd-like tuning, which suggests the computational goal of MSTd may not be to accurately estimate self-motion. We found that adding computational constraints inspired by an existing MSTd model that performs dimensionality reduction on afferent motion signals improves the correspondence with MSTd.

neuroscience↗

bioRxiv2021

Decoding Estimates of Curvilinear Self-Motion from Neural Signals in a Model of Primate MSTd

Self-motion produces characteristic patterns of optic flow on the eye of the mobile observer. Movement along linear, straight paths without eye movements yields motion that radiates from the direction of travel (heading). The observer experiences more complex motion patterns while moving along more general curvilinear (e.g. circular) paths, the appearance of which depends on the radius of the curved path (path curvature) and the direction of gaze. Neurons in brain area MSTd of primate visual cortex exhibit tuning to radial motion patterns and have been linked with linear heading perception. MSTd also contains neurons that exhibit tuning to spirals, but their function is not well understood. We investigated in a computational model whether MSTd, through its diverse pattern tuning, could support estimation of a broader range of self-motion parameters from optic flow than has been previously demonstrated. We used deep learning to decode these parameters from signals produced by neurons tuned to radial expansion, spiral, ground flow, and other patterns in a mechanistic neural model of MSTd. Specifically, we found that we could accurately decode the clockwise/counterclockwise sign of curvilinear path and the gaze direction relative to the path tangent from spiral cells; heading from radial cells; and the curvature (radius) of the curvilinear path from activation produced by both radial and spiral populations. We demonstrate accurate decoding of these linear and curvilinear self-motion parameters in both synthetic and naturalistic videos of simulated self-motion. Estimates remained stable over time, while also rapidly adapting to dynamic changes in the observers curvilinear self-motion. Our findings suggest that specific populations of neurons in MSTd could effectively signal important aspects of the observers linear and curvilinear self-motion. Author SummaryHow do we perceive our self-motion as we move through the world? Substantial evidence indicates that brain area MSTd contains neurons that signal the direction of travel during movement along straight paths. We wondered whether MSTd neurons could also estimate more general self-motion along curved paths. We tested this idea by using deep learning to decode signals produced by a neural model of MSTd. The system accurately decoded parameters that specify the observers self-motion along straight and curved paths in videos of synthetic and naturalistic scenes rendered in the Unreal game engine. Our findings suggest that MSTd could jointly signal self-motion along straight and curved paths.

neuroscience↗

bioRxiv2021

A Dynamic Efficient Sensory Encoding Approach to Adaptive Tuning in Neural Models of Visual Motion Processing

This paper introduces a self-tuning mechanism for capturing rapid adaptation to changing visual stimuli by a population of neurons. Building upon the principles of efficient sensory encoding, we show how neural tuning curve parameters can be continually updated to optimally encode a time-varying distribution of recently detected stimulus values. We implemented this mechanism in a neural model that produces human-like estimates of self-motion direction (i.e., heading) based on optic flow. The parameters of speed-sensitive units were dynamically tuned in accordance with efficient sensory encoding such that the network remained sensitive as the distribution of optic flow speeds varied. In two simulation experiments, we found that model performance with dynamic tuning yielded more accurate, shorter latency heading estimates compared to the model with static tuning. We conclude that dynamic efficient sensory encoding offers a plausible approach for capturing adaptation to varying visual environments in biological visual systems and neural models alike.

neuroscience↗