Biology subjects

Whiteway, M. R.

Publications and source records attributed to Whiteway, M. R..

12 recordsLinked to original sources

bioRxiv2026

Lightning Pose 3D: an uncertainty-aware framework for data-efficient multi-view animal pose estimation

Multi-view pose estimation is essential for quantifying animal behavior in scientific research, yet current methods struggle to achieve accurate tracking with limited labeled data and suffer from poor uncertainty estimates. We address these challenges with a flexible framework that can operate with or without camera calibration, combining novel training and post-processing techniques with an uncertainty-aware pseudo-labeling distillation procedure. Our multi-view model processes all camera views jointly using a pretrained vision transformer backbone, and a simulated occlusion technique encourages the model to learn robust cross-view correspondences without requiring camera parameters. When camera parameters are available, 3D data augmentations and a triangulation-based loss further encourage geometric consistency. We extend the Ensemble Kalman Smoother (EKS) post-processor to the nonlinear case, leveraging camera geometry, and introduce a variance inflation technique that detects cross-view inconsistencies and corrects overconfident predictions. We validate our approach on five datasets spanning three species (fly, mouse, bird), including a multi-animal dataset with two visually distinct individuals; the proposed pipeline consistently outperforms existing methods across datasets. We demonstrate how these improvements translate to downstream scientific analyses using data from the International Brain Laboratory, showing improved unsupervised behavioral clustering and neural decoding of paw kinematics with just 200 labeled frames. To facilitate adoption, we developed a browser-based, cloud-compatible user interface that supports the full life cycle of multi-view pose estimation, from labeling and model training to post-processing with EKS and diagnostic visualizations.

neuroscience↗

bioRxiv2025

Asymmetric Social Representations in the Prefrontal Cortex for Cooperative Behavior

Cooperation is a hallmark of social species, enabling individuals to achieve goals that are unattainable alone. Across species, cooperative behaviors are often organized by distinct social roles such as leaders and followers, yet the neural mechanisms supporting such role-based coordination remain elusive. Here we introduce a new paradigm for studying cooperation in mice, where pairs of animals engage in a joint spatial foraging task that naturally gives rise to stable leader-follower roles predictive of learning speed. Disruption of medial prefrontal cortex (mPFC) activity, particularly in followers, impairs cooperation and induces reciprocal shifts in how animals weigh self- and partner-related cues for decision-making. Calcium imaging reveals that mPFC encodes both leadership dynamics and an egocentric social value map of the partners position, each in an asymmetric, role-specific manner. Combining this behavior with a novel multi-agent inverse reinforcement learning framework, we identify latent value functions that guide cooperative decisions and are decodable from mPFC activity. These findings uncover fundamental neural computations that support cooperation, revealing how social roles shape decision-making in real time. Our work opens new avenues for investigating the cellular and circuit basis of social cognition and collective behavior.

neuroscience↗

bioRxiv2024

Coordination and persistence of aggressive visual communication in Siamese fighting fish

Animals coordinate their behavior with each other during both cooperative and agonistic social interactions. Such coordination often adopts the form of "turn taking", in which the interactive partners alternate the performance of a behavior. Apart from acoustic communication, how turn taking between animals is coordinated is not well understood. Furthermore, the neural substrates that regulate persistence in engaging in social interactions are poorly studied. Here, we use Siamese fighting fish (Betta splendens), to study visually-driven turn-taking aggressive behavior. Using encounters with conspecifics and with animations, we characterize the dynamic visual features of an opponent and the behavioral sequences that drive turn taking. Through a brain-wide screen of neuronal activity during coordinated and persistent aggressive behavior, followed by targeted brain lesions, we find that the caudal portion of the dorsomedial telencephalon, an amygdala-like region, promotes persistent participation in aggressive interactions, yet is not necessary for coordination. Our work highlights how dynamic visual cues shape the rhythm of social interactions at multiple timescales, and points to the pallial amygdala as a region controlling engagement in such interactions. These results suggest an evolutionarily conserved role of the vertebrate pallial amygdala in regulating the persistence of emotional states.

neuroscience↗

bioRxiv2023

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

neuroscience↗

bioRxiv2021

Flygenvectors: The spatial and temporal structure of neural activity across the fly brain

What are the spatial and temporal scales of brainwide neuronal activity, and how do activities at different scales interact? We used SCAPE microscopy to image a large fraction of the central brain of adult Drosophila melanogaster with high spatiotemporal resolution while flies engaged in a variety of behaviors, including running, grooming and flailing. This revealed neural representations of behavior on multiple spatial and temporal scales. The activity of most neurons across the brain correlated (or, in some cases, anticorrelated) with running and flailing over timescales that ranged from seconds to almost a minute. Grooming elicited a much weaker global response. Although these behaviors accounted for a large fraction of neural activity, residual activity not directly correlated with behavior was high dimensional. Many dimensions of the residual activity reflect the activity of small clusters of spatially organized neurons that may correspond to genetically defined cell types. These clusters participate in the global dynamics, indicating that neural activity reflects a combination of local and broadly distributed components. This suggests that microcircuits with highly specified functions are provided with knowledge of the larger context in which they operate, conferring a useful balance of specificity and flexibility.

neuroscience↗

bioRxiv2021

Semi-supervised sequence modeling for improved behavioral segmentation

A popular approach to quantifying animal behavior from video data is through discrete behavioral segmentation, wherein video frames are labeled as containing one or more behavior classes such as walking or grooming. Sequence models learn to map behavioral features extracted from video frames to discrete behaviors, and both supervised and unsupervised methods are common. However, each approach has its drawbacks: supervised models require a time-consuming annotation step where humans must hand label the desired behaviors; unsupervised models may fail to accurately segment particular behaviors of interest. We introduce a semi-supervised approach that addresses these challenges by constructing a sequence model loss function with (1) a standard supervised loss that classifies a sparse set of hand labels; (2) a weakly supervised loss that classifies a set of easy-to-compute heuristic labels; and (3) a self-supervised loss that predicts the evolution of the behavioral features. With this approach, we show that a large number of unlabeled frames can improve supervised segmentation in the regime of sparse hand labels and also show that a small number of hand labeled frames can increase the precision of unsupervised segmentation.

animal behavior and cognition↗

bioRxiv2021

Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders

Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.

neuroscience↗

bioRxiv2020

Recurrent Switching Dynamical Systems Models for Multiple Interacting Neural Populations

Modern recording techniques can generate large-scale measurements of multiple neural populations over extended time periods. However, it remains a challenge to model non-stationary interactions between high-dimensional populations of neurons. To tackle this challenge, we develop recurrent switching linear dynamical systems models for multiple populations. Here, each high-dimensional neural population is represented by a unique set of latent variables, which evolve dynamically in time. Populations interact with each other through this low-dimensional space. We allow the nature of these interactions to change over time by using a discrete set of dynamical states. Additionally, we parameterize these discrete state transition rules to capture which neural populations are responsible for switching between interaction states. To fit the model, we use variational expectation-maximization with a structured mean-field approximation. After validating the model on simulations, we apply it to two different neural datasets: spiking activity from motor areas in a non-human primate, and calcium imaging from neurons in the nematode C. elegans. In both datasets, the model reveals behaviorally-relevant discrete states with unique inter-population interactions and different populations that predict transitioning between these states.

neuroscience↗

bioRxiv2020

Deep Graph Pose: a semi-supervised deep graphical model for improved animal pose tracking

Noninvasive behavioral tracking of animals is crucial for many scientific investigations. Recent transfer learning approaches for behavioral tracking have considerably advanced the state of the art. Typically these methods treat each video frame and each object to be tracked independently. In this work, we improve on these methods (particularly in the regime of few training labels) by leveraging the rich spatiotemporal structures pervasive in behavioral video -- specifically, the spatial statistics imposed by physical constraints (e.g., paw to elbow distance), and the temporal statistics imposed by smoothness from frame to frame. We propose a probabilistic graphical model built on top of deep neural networks, Deep Graph Pose (DGP), to leverage these useful spatial and temporal constraints, and develop an efficient structured variational approach to perform inference in this model. The resulting semi-supervised model exploits both labeled and unlabeled frames to achieve significantly more accurate and robust tracking while requiring users to label fewer training frames. In turn, these tracking improvements enhance performance on downstream applications, including robust unsupervised segmentation of behavioral "syllables," and estimation of interpretable "disentangled" low-dimensional representations of the full behavioral video. Open source code is available at https://github.com/paninski-lab/deepgraphpose.

animal behavior and cognition↗

bioRxiv2020

A latent variable approach to decoding neural population activity

Decoding is a powerful approach for measuring the information contained in the activity of neural populations. As a result, decoding analyses are now used across a wide range of model organisms and experimental paradigms. However, typical analyses employ general purpose decoding algorithms that do not explicitly take advantage of the structure of neural variability, which is often low-dimensional and can thus be effectively characterized using latent variables. Here we propose a new decoding framework that exploits the low-dimensional structure of neural population variability by removing correlated variability that is unrelated to the decoded variable, then decoding the resulting denoised activity. We demonstrate the efficacy of this framework using simulated data, where the true upper bounds for decoding performance are known. A linear version of our decoder provides an estimator for the decoded variable that can be more efficient than other commonly used linear estimators such as linear discriminant analysis. In addition, our proposed decoding framework admits a simple extension to nonlinear decoding that compares favorably to standard feed-forward neural networks. By explicitly modeling shared population variability, the success of the resulting linear and nonlinear decoders also offers a new perspective on the relationship between shared variability and information contained in large neural populations.

neuroscience↗

bioRxiv2018

Characterizing the nonlinear structure of shared variability in cortical neuron populations using neural networks

Sensory neurons often have variable responses to repeated presentations of the same stimulus, which can significantly degrade the information contained in those responses. Such variability is often shared across many neurons, which in principle can allow a decoder to mitigate the effects of such noise, depending on the structure of the shared variability and its relationship to sensory encoding at the population level. Latent variable models offer an approach for characterizing the structure of this shared variability in neural population recordings, although they have thus far typically been used under restrictive mathematical assumptions, such as assuming linear transformations between the latent variables and neural activity. Here we leverage recent advances in machine learning to introduce two nonlinear latent variable models for analyzing large-scale neural recordings. We first present a general nonlinear latent variable model that is agnostic to the stimulus tuning properties of the individual neurons, and is hence well suited for exploring neural populations whose tuning properties are not well characterized. This motivates a second class of model, the Generalized Affine Model, which simultaneously determines each neurons stimulus selectivity and a set of latent variables that modulate these stimulus responses both additively and multiplicatively. While these approaches can detect general nonlinear relationships in shared neural variability, we find that neural activity recorded in anesthetized primary visual cortex (V1) is best described by a single additive and single multiplicative latent variable, i.e., an \"affine model\". In contrast, application of the same models to recordings in awake macaque prefrontal cortex discover more general nonlinearities to compactly describe the population response variability. These results thus demonstrate how nonlinear latent variable models can be used to describe population variability, and suggest that a range of methods is necessary to study different brain regions under different experimental conditions.

neuroscience↗

bioRxiv2017

Spatial organization of the mouse auditory cortex to sound dynamics revealed using automated image segmentation

Sound stimuli are characterized by their rich spectral and temporal dynamic properties. Individual neurons in auditory cortex (ACX) encode both spectral and temporal aspects of sounds e.g. sound onset and/or offset. While the different fields of the ACX show gradients of frequency selectivity the large-scale organization of sound dynamics is unknown. We used widefield imaging of GCaMP6s in awake mouse ACX combined with a novel unsupervised image segmentation technique to investigate the spatiotemporal representation of sound onset and offset. Using this technique, we identified known auditory fields but also detected novel ACX areas. Furthermore, we found that ACX areas differed in their responses to tone onset and offset. Multiple areas were preferentially activated by tone offset, and on-response areas were more spatially localized than off-response areas. We also found tonotopy in off-responses. Together our results demonstrate a different spatial distribution of neurons across ACX for processing sound onsets versus offsets.

neuroscience↗