Biology subjects

Simoncelli, E. P.

Publications and source records attributed to Simoncelli, E. P..

14 recordsLinked to original sources

bioRxiv2024

Foundations of visual form selectivity for neurons in macaque V1 and V2

Neurons early in the primate visual cortical pathway generate responses by combining signals from other neurons: some from downstream areas, some from within the same area, and others from areas upstream. Here we develop a model that selectively combines afferents derived from a population model of V1 cells. We use this model to account for responses we recorded of both V1 and V2 neurons in awake fixating macaque monkeys to stimuli composed of a sparse collection of locally oriented features ("droplets") designed to drive subsets of V1 neurons. The first stage computes the rectified responses of a fixed population of oriented filters at different scales that cover the visual field. The second stage computes a weighted combination of these first-stage responses, followed by a final nonlinearity, with parameters optimized to fit data from physiological recordings and constrained to encourage sparsity and locality. The fitted model accounts for the responses of both V1 and V2 neurons, capturing an average of 43% of the explainable variance for V1 and 38% for V2. The models fitted to droplet recordings predict responses to classical stimuli, such as gratings of different orientations and spatial frequencies, as well as to textures of different spectral content, which are known to be especially effective in driving V2. The models are less effective, however, at capturing the selectivity of responses to textures that include naturalistic image statistics. The pattern of afferents -- defined by their weights over the 4 dimensions of spatial position, orientation, and spatial frequency -- provides a common and interpretable characterization of the origin of many neuronal response properties in the early visual cortex.

neuroscience↗

bioRxiv2024

Neuronal and behavioral responses to naturalistic texture images in macaque monkeys

The visual world is richly adorned with texture, which can serve to delineate important elements of natural scenes. In anesthetized macaque monkeys, selectivity for the statistical features of natural texture is weak in V1, but substantial in V2, suggesting that neuronal activity in V2 might directly support texture perception. To test this, we investigated the relation between single cell activity in macaque V1 and V2 and simultaneously measured behavioral judgments of texture. We generated stimuli along a continuum between naturalistic texture and phase-randomized noise and trained two macaque monkeys to judge whether a sample texture more closely resembled one or the other extreme. Analysis of responses revealed that individual V1 and V2 neurons carried much less information about texture naturalness than behavioral reports. However, the sensitivity of V2 neurons, especially those preferring naturalistic textures, was significantly closer to that of behavior compared with V1. The firing of both V1 and V2 neurons predicted perceptual choices in response to repeated presentations of the same ambiguous stimulus in one monkey, despite low individual neural sensitivity. However, neither population predicted choice in the second monkey. We conclude that neural responses supporting texture perception likely continue to develop downstream of V2. Further, combined with neural data recorded while the same two monkeys performed an orientation discrimination task, our results demonstrate that choice-correlated neural activity in early sensory cortex is unstable across observers and tasks, untethered from neuronal sensitivity, and thus unlikely to reflect a critical aspect of the formation of perceptual decisions. Significance statementAs visual signals propagate along the cortical hierarchy, they encode increasingly complex aspects of the sensory environment and likely have a more direct relationship with perceptual experience. We replicate and extend previous results from anesthetized monkeys differentiating the selectivity of neurons along the first step in cortical vision from area V1 to V2. However, our results further complicate efforts to establish neural signatures that reveal the relationship between perception and the neuronal activity of sensory populations. We find that choice-correlated activity in V1 and V2 is unstable across different observers and tasks, and also untethered from neuronal sensitivity and other features of nonsensory response modulation.

neuroscience↗

bioRxiv2024

Responses of neurons in macaque V4 to object and texture images

Humans and monkeys can rapidly recognize objects in everyday scenes. It is not fully understood where this computation arises, but previous work suggests selectivity for object shape first emerges in cortical area V4. To explore the mechanisms of this selectivity, we generated images on a continuum between "scrambled" textures and photographic images, preserving the local statistics of the original image while discarding structural information about scene and shape. We defined this continuum by the size of the region in which statistics were measured ("pooling region"). On average, neurons responded equally to photographic images and their scrambled counterparts. However, neurons exhibited a greater dynamic range of response to photographs. As a result, neuronal populations in V4 could more reliably discriminate between photographs than between scrambled images. Responses to partially scrambled images were more similar to responses to fully scrambled images than to photographs, even for perceptually subtle changes, and only began to resemble responses to photographs for pooling regions roughly half the size of typical V4 receptive fields. These same patterns emerged in an image similarity metric designed to predict human judgements of image degradation. Finally, V4 object selectivity showed dynamics that grew slowly and persisted following response offset, suggesting this signal may arise from recurrent mechanisms. Significance StatementObject recognition is a primary goal of visual processing, but is too complex to be computed by early visual areas. To evaluate where object-selective signals emerge, we recorded the responses of individual neurons in mid-level area V4 as macaque monkeys viewed images of objects ("things") and of matched textures ("stuff"). Object selective V4 responses emerged in two phases. Short-latency responses discriminate objects from one another, but the distinction between objects and textures arrived later. This late-emerging signal was sensitive even to small image degradations, as is human perception. Our results show how V4 initiates the brains representation of visual objects.

neuroscience↗

bioRxiv2023

Comparing neural models using their perceptual discriminability predictions

Internal representations are not uniquely identifiable from perceptual measurements: different representations can generate identical perceptual predictions, and similar representations may predict dissimilar percepts. Here, we generalize a previous method ("Eigendistortions" - Berardino et al., 2017) to enable comparison of models based on their metric tensors, which can be verified perceptually. Metric tensors characterize sensitivity to stimulus perturbations, reflecting both the geometric and stochastic properties of the representation, and providing an explicit prediction of perceptual discriminability. Brute force comparison of model-predicted metric tensors would require estimation of human perceptual thresholds along an infeasibly large set of stimulus directions. To circumvent this "perceptual curse of dimensionality", we compute and measure discrimination capabilities for a small set of most-informative perturbations, reducing the measurement cost from thousands of hours (a conservative estimate) to a single trial. We show that this single measurement, made for a variety of different test stimuli, is sufficient to differentiate models, select models that better match human perception, or generate new models that combine the advantages of existing models. We demonstrate the power of this method in comparison of (1) two models for trichromatic color representation, with differing internal noise; and (2) two autoencoders trained with different regularizers.

neuroscience↗

bioRxiv2023

Fixational Eye Movements Enhance the Precision of Visual Information Transmitted by the Primate Retina

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulusevoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

neuroscience↗

bioRxiv2022

Perceptual learning improves discrimination while distorting appearance

Perceptual sensitivity often improves with training, a phenomenon known as perceptual learning. Another important perceptual dimension is appearance, the subjective sense of stimulus magnitude. Are training-induced improvements in sensitivity accompanied by more accurate appearance? Here, we examine this question by measuring both discrimination and estimation capabilities for near-horizontal motion perception, before and after training. Human observers who trained on either discrimination or estimation exhibited improvements in discrimination accuracy alongside increased biases in their estimates away from horizontal. To explain this counterintuitive finding, we developed a computational observer model in which perceptual learning arises from increases in the precision of underlying neural representations. For each observer, the fitted model accounted for both discrimination performance and the distribution of estimates, and their changes after training. Our empirical findings and modeling suggest that learning enhances distinctions between categories, a potentially important aspect of real-world perception and perceptual learning.

neuroscience↗

bioRxiv2022

Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors

Visual information arriving at the retina is transmitted to the brain by signals in the optic nerve, and the brain must rely solely on these signals to make inferences about the visual world. Previous work has probed the content of these signals by directly reconstructing images from retinal activity using linear regression or nonlinear regression with neural networks. Maximum a posteriori (MAP) reconstruction using retinal encoding models and separately-trained natural image priors offers a more general and principled approach. We develop a novel method for approximate MAP reconstruction that combines a generalized linear model for retinal responses to light, including their dependence on spike history and spikes of neighboring cells, with the image prior implicitly embedded in a deep convolutional neural network trained for image denoising. We use this method to reconstruct natural images from ex vivo simultaneously-recorded spikes of hundreds of retinal ganglion cells uniformly sampling a region of the retina. The method produces reconstructions that match or exceed the state-of-the-art in perceptual similarity and exhibit additional fine detail, while using substantially fewer model parameters than previous approaches. The use of more rudimentary encoding models (a linear-nonlinear-Poisson cascade) or image priors (a 1/f spectral model) significantly reduces reconstruction performance, indicating the essential role of both components in achieving high-quality reconstructed images from the retinal signal.

neuroscience↗

bioRxiv2022

A common framework for discriminability and perceived intensity of sensory stimuli

The perception of sensory attributes is often quantified through measurements of sensitivity (the ability to detect small stimulus changes), as well as through direct judgements of appearance or intensity. Despite their ubiquity, the relationship between these two measurements remains controversial and unresolved. Here, we propose a framework in which they arise from different aspects of a common representation. Specifically, we assume that judgements of stimulus intensity (e.g., as measured through rating scales) reflect the mean value of an internal representation, and sensitivity reflects a combination of mean value and noise properties, as quantified by the statistical measure of Fisher Information. Unique identification of these internal representation properties can be achieved by combining measurements of sensitivity and judgments of intensity. As a central example, we show that Webers law of perceptual sensitivity can co-exist with Stevens power-law scaling of intensity ratings (for all exponents), when the noise amplitude increases in proportion to the representational mean. We then extend this result beyond the Webers law range by incorporating a more general and physiology-inspired form of noise, and show that the combination of noise properties and sensitivity measurements accurately predicts intensity ratings across a variety of sensory modalities and attributes. Our framework unifies two primary perceptual measurements - thresholds for sensitivity and rating scales for intensity - and provides a neural interpretation for the underlying representation. Significance StatementPerceptual measurements of sensitivity to stimulus changes and stimulus appearance (intensity) are ubiquitous in the study of perception. However, the relationship between these two seemingly disparate measurements remains unclear. Proposals for unification have been made for over 60 years, but they generally lack support from perceptual or physiological measurements. Here, we provide a framework that offers a unified interpretation of perceptual sensitivity and intensity measurements, and we demonstrate its consistency with experimental measurements across multiple perceptual domains.

neuroscience↗

bioRxiv2021

Mapping Spatial Frequency Preferences Across Human Primary Visual Cortex

AO_SCPLOWBSTRACTC_SCPLOWNeurons in primate visual cortex (area V1) are tuned for spatial frequency, in a manner that depends on their position in the visual field. Several studies have examined this dependency using fMRI, reporting preferred spatial frequencies (tuning curve peaks) of V1 voxels as a function of eccentricity, but their results differ by as much as two octaves, presumably due to differences in stimuli, measurements, and analysis methodology. Here, we characterize spatial frequency tuning at a millimeter resolution within human primary visual cortex, across stimulus orientation and visual field locations. We measured fMRI responses to a novel set of stimuli, constructed as sinusoidal gratings in log-polar coordinates, which include circular, radial, and spiral geometries. For each individual stimulus, the local spatial frequency varies inversely with eccentricity, and for any given location in the visual field, the full set of stimuli span a broad range of spatial frequencies and orientations. Over the measured range of eccentricities, the preferred spatial frequency is well-fit by a function that varies as the inverse of the eccentricity plus a small constant. We also find small but systematic effects of local stimulus orientation, defined in both absolute coordinates and relative to visual field location. Specifically, peak spatial frequency is higher for pinwheel than annular stimuli and for horizontal than vertical stimuli.

neuroscience↗

bioRxiv2021

Targeted comodulation supports flexible and accurate decoding in V1

Sensory-guided behavior requires reliable encoding of stimulus information in neural populations, and task-specific readout through a selective combination of these responses. The former has been the topic of extensive study, but the means by which the brain achieves the latter remain poorly understood, especially when adapting to changing task demands. Here we introduce a novel theory for adaptive sensory processing based on functionally-targeted stochastic modulation. We find that responses of neurons in area V1 of monkeys performing a visual orientation discrimination task exhibit low-dimensional, rapidly fluctuating gain modulation, which is stronger in neurons that are most informative for the current behavioral task. We propose that this modulation serves as a label that supports adaptive downstream readout. Our theoretical and experimental results show that V1 modulation can be used to decode from neural activity after only a small number of training trials, consistent with observed behavior. In a hierarchical visual neural network model, the modulator-induced labels are learned quickly and accompany task information across several processing stages to finally guide decisions. Consistent with this model, we find that the V1 modulatory signal is also present in the activity of simultaneously recorded MT units, and that its label of task information is preserved. Our findings provide evidence for a novel mechanism for task-adaptive information routing in the brain, through targeted co-modulation.

neuroscience↗

bioRxiv2020

Pinpointing the neural signatures of single-exposure visual familiarity

Memories of the images that we have seen are thought to be reflected in the reduction of neural responses in high-level visual areas such as inferotemporal (IT) cortex, a phenomenon known as repetition suppression (RS). We challenged this hypothesis with a task that required rhesus monkeys to report whether images were novel or repeated while ignoring variations in contrast, a stimulus attribute that is also known to modulate the overall IT response. The monkeys behavior was largely contrast-invariant, contrary to the predictions of an RS-inspired decoder, which could not distinguish responses to images that are repeated from those of lower contrast. However, the monkeys behavioral patterns were well-predicted by a linearly decodable variant in which the total spike count was corrected for contrast modulation. These results suggest that the IT neural activity pattern that best aligns with single-exposure visual recognition memory behavior is not RS but rather "sensory referenced suppression (SRS)": reductions in IT population response magnitude, corrected for sensory modulation. Significance statementMemories of whether an image has been seen before are reflected in high-level visual cortex as "sensory referenced suppression (SRS)": reductions in population response magnitude, corrected for sensory modulation.

neuroscience↗

bioRxiv2020

Learning efficient task-dependent representations with synaptic plasticity

Neural populations do not perfectly encode the sensory world: their capacity is limited by the number of neurons, metabolic and other biophysical resources, and intrinsic noise. The brain is presumably shaped by these limitations, improving efficiency by discarding some aspects of incoming sensory streams, while prefer-entially preserving commonly occurring, behaviorally-relevant information. Here we construct a stochastic recurrent neural circuit model that can learn efficient, task-specific sensory codes using a novel form of reward-modulated Hebbian synaptic plasticity. We illustrate the flexibility of the model by training an initially unstructured neural network to solve two different tasks: stimulus estimation, and stimulus discrimination. The network achieves high performance in both tasks by appropriately allocating resources and using its recurrent circuitry to best compensate for different levels of noise. We also show how the interaction between stimulus priors and task structure dictates the emergent network representations.

neuroscience↗

bioRxiv2019

Flexible and accurate decoding of neural populations through stochastic comodulation

Sensory-guided behavior requires reliable encoding of information (from stimuli to neural responses) and flexible decoding (from neural responses to behavior). In typical decision tasks, a small subset of cells within a large population encode task-relevant stimulus information and need to be identified by later processing stages for relevant information to be transmitted. A statistically optimal decoder (e.g., maximum likelihood) can utilize task-relevant cells for any given task configuration, but relies on complete knowledge of the relationship between the task and the stimulus-response and noise properties of the encoding population. The brain could learn an optimal decoder for a task through supervised learning (i.e., regression), but this typically requires many training trials, and thus lacks the flexibility of humans or animals, that can rapidly adjust to changes in task parameters or structure. Here, we propose a novel decoding solution based on functionally targeted stochastic modulation. Population recordings during different discrimination tasks have revealed that a substantial portion of trial-to-trial variability in cell responses can be explained by stochastic modulatory signals that are shared, and that seem to preferentially target task-informative neurons (Rabinowitz et al., 2015). The variability introduced by these modulators corrupts the encoded stimulus signal, but we propose that it also serves as a label for the informative neurons, allowing the decoder to solve the identification problem. We show in simulations of a modulated Poisson spiking model that a linear decoder with readout weights proportional to the estimated neuron-specific strength of modulation achieves performance close to an optimal decoder.

neuroscience↗

bioRxiv2018

Inference of Nonlinear Spatial Subunits by Spike-Triggered Clustering in Primate Retina

Integration of rectified synaptic inputs is a widespread nonlinear motif in sensory neuroscience. We present a novel method for maximum likelihood estimation of nonlinear subunits by soft-clustering spike-triggered stimuli. Subunits estimated from parasol ganglion cells recorded in macaque retina partitioned the receptive field into compact regions, likely representing bipolar cell inputs. Joint clustering with multiple RGCs revealed shared subunits in neighboring cells, producing a parsimonious population model. Closed-loop subunit validation was then performed by projecting white noise into the null space of the linear receptive field. Responses to these null stimuli were more accurately explained by a model with multiple subunits, and were stronger in OFF cells than ON cells. Presentation of natural stimuli containing jittering edges and textures also revealed greater response prediction accuracy with the subunit model. Finally, the generality of the approach was demonstrated by application to V1 data.

neuroscience↗