Biology subjects

Mathis, A.

Publications and source records attributed to Mathis, A..

17 recordsLinked to original sources

bioRxiv2026

Closed-loop imitation learning reveals muscle-centric and latent-goal codes in primate sensorimotor cortex

Dexterous grasping requires the seamless integration of proprioceptive feedback with predictive motor commands. Yet, how cortical circuits combine afferent feedback with efference copies to support skilled hand control remains poorly understood. Here we develop a closed-loop, muscle-level model of primate grasping that integrates biomechanics, imitation learning, and neural recordings. A neural network policy trained on a 39-muscle musculoskeletal hand reproduces naturalistic pre-contact shaping and develops internal states that quantitatively explain single-neuron activity in primary motor (M1) and somatosensory (S1) cortices. Three principles emerged. First, muscle-based controllers generate representations that align more closely with cortical dynamics than joint-based controllers, despite lower kinematic accuracy. Second, recurrent architectures with temporal memory, especially LSTMs, provide an inductive bias that enhances neural predictability. Third, model-to-brain alignment peaked at the layer integrating proprioceptive and goal signals. Finally, by decoding the models latent trajectory representation from M1, we demonstrated direct neural control of the policy: with activity from only tens of neurons, the brain-driven controller generated coherent grasp trajectories and showed markedly greater robustness to noise than joint-angle decoding. These findings reveal that S1 and M1 embed integrated, temporally structured, muscle-centric states and establish a stimulus-computable mechanistic framework for modeling sensorimotor control, while opening a novel route for creating brain-body models.

neuroscience↗

bioRxiv2025

DLC2Action: A Deep Learning-based Toolbox for Automated Behavior Segmentation

Behavioral analysis is fundamental to neuroscience, yet the manual annotation of actions remains a bottleneck that constrains both the scale and the reproducibility of experiment. Here, we present DLC2Action, an open-source Python toolbox that enables automatic behavior annotation from video, audio and estimated 2D/3D pose tracking data. DLC2Action integrates multiple state-of-the-art deep learning architectures optimized for action segmentation and supports self-supervised learning (SSL) to address annotation scarcity, boosting performance with limited labeled datasets. To enable model comparison, we establish fixed train/test partitions for nine diverse datasets comprising rodent experiments, human cooking studies, and wildlife observation. DLC2Action reached strong performance across those benchmarks. To further showcase the tools versatility, we applied it to Atari gameplay data and found that, in certain games, the players eye movements consistently predict their button presses across subjects. Because DLC2Action features an intuitive graphical user interface (GUI), users can streamline behavior annotation, perform active learning, and assess of model predictions. Diverse pose, video, and annotation formats are supported. Lastly, DLC2Action is modular and thus designed for extensibility, allowing users to integrate new models, dataset features, and methods. The code and benchmarks are available at: https://github.com/amathislab/DLC2action

animal behavior and cognition↗

bioRxiv2025

Sensory Compression as a Unifying Principle for Action Chunking and Time Coding in the Brain

The dorsolateral striatum (DLS) supports diverse time-sensitive behaviors--action chunking, duration estimation, and motor timing--yet no single framework is able to explain all of these phenomena. Here, we propose that the massive convergence of cortical projections onto the striatum provides such a framework. We develop a corticostriatal neural network model in which a recurrent cortical module communicates with a recurrent striatal module through a low-dimensional, noisy bottleneck, with the whole system trained via reinforcement learning. Across three DLS-associated tasks, compression produces a consistent computational motif whereby cortex provides low-dimensional control signals while the striatum generates stable, time-encoding dynamics. This separation gives rise to chunking behavior with action slipping, intensity-biased duration judgements with stimulus-modulated time coding, and stereotyped motor timing programs. Perturbation of the compressed cortical signal causally shifts behavior while preserving sequential structure in striatum activity. In sum, our results unify information-theoretic and dynamical systems perspectives of basal ganglia function to link anatomical compression to a variety of temporally-sensitive sensorimotor behaviors implicated in the DLS.

neuroscience↗

bioRxiv2025

Mice navigate scent trails using predictive policies

Many terrestrial animals accurately follow odor trails, but their navigation strategies remain unclear. We presented mice with dynamic, non-repeating odor trails using a paper treadmill, and found that they rapidly learned to track trails effectively. Odor detections by one or both nostrils during inhalation evoked rapid, stereotyped motor responses. Blocking one nostril led to biased tracking, while disconnecting interhemispheric neural communication disrupted tracking almost entirely. We found that trail tracking is more than a reflex, with mice adapting their movements according to a memory of recently encountered trail geometry. A Bayesian model captures these behaviors, suggesting mice combine bilateral olfactory input with short-term memory to guide navigation actively. Functional perturbations and neural recordings from behaving animals implicate an early olfactory cortical region in trail tracking. These results lay the groundwork to uncover the computational architecture of sensorimotor integration in naturalistic tasks.

neuroscience↗

bioRxiv2025

Deep-learning models of the ascending proprioceptive pathway are subject to illusions

Proprioception is essential for perception and action. Like any other sense, proprioception is also subject to illusions. In this study, we model classic proprioceptive illusions in which tendon vibrations lead to biases in estimating the state of the body. We investigate these illusions with task-driven models that have been trained to infer the state of the body from distributed sensory muscle spindle inputs (primary and secondary afferents). Recent work has shown that such models exhibit representations similar to the neural code along the ascending proprioceptive pathway. Importantly, we did not train the models on illusion experiments and simulated muscle-tendon vibrations by considering their effect on primary afferents. Our results demonstrate that task-driven models are indeed susceptible to proprioceptive illusions, with the magnitude of the illusion depending on the vibration frequency. This work illustrates that primary afferents alone are sufficient to account for these classic illusions and provides a foundation for future theory-driven experiments.

neuroscience↗

bioRxiv2024

Modeling Sensorimotor Processing with Physics-Informed Neural Networks

Proprioception is essential for planning and executing precise movements. Muscle spindles, the key mechanoreceptors for proprioception, are the principle sensory neurons enabling this process. Emerging evidence suggests spindles act as adaptable processors, modulated by gamma motor neurons to meet task demands. Yet, the specifics of this modulation remain unknown. Here, we present a novel, physics-informed neural network model that integrates biomechanics and neural dynamics to capture spindle function with high fidelity and efficiency, while maintaining computational tractability. Through validation across multiple experimental datasets and species, our model not only outperforms existing approaches but also reveals key drivers of variability in spindle responses, offering new insights into proprioceptive mechanisms.

neuroscience↗

bioRxiv2024

Elucidating the Hierarchical Nature of Behavior with Masked Autoencoders

Natural behavior is hierarchical. Yet, there is a paucity of benchmarks addressing this aspect. Recognizing the scarcity of large-scale hierarchical behavioral benchmarks, we create a novel synthetic basketball playing benchmark (Shot7M2). Beyond synthetic data, we extend BABEL into a hierarchical action segmentation benchmark (hBABEL). Then, we develop a masked autoencoder framework (hBehaveMAE) to elucidate the hierarchical nature of motion capture data in an unsupervised fashion. We find that hBehaveMAE learns interpretable latents on Shot7M2 and hBABEL, where lower encoder levels show a superior ability to represent fine-grained movements, while higher encoder levels capture complex actions and activities. Additionally, we evaluate hBehaveMAE on MABe22, a representation learning benchmark with short and long-term behavioral states. hBehaveMAE achieves state-of-the-art performance without domain-specific feature extraction. Together, these components synergistically contribute towards unveiling the hierarchical organization of natural behavior. Models and benchmarks are available at https://github.com/amathislab/BehaveMAE.

animal behavior and cognition↗

bioRxiv2024

Acquiring musculoskeletal skills with curriculum-based reinforcement learning

Efficient musculoskeletal simulators and powerful learning algorithms provide computational tools to tackle the grand challenge of understanding biological motor control. Our winning solution for the inaugural NeurIPS MyoChallenge leverages an approach mirroring human skill learning. Using a novel curriculum learning approach, we trained a recurrent neural network to control a realistic model of the human hand with 39 muscles to rotate two Baoding balls in the palm of the hand. In agreement with data from human subjects, the policy uncovers a small number of kinematic synergies even though it is not explicitly biased towards low-dimensional solutions. However, by selectively inactivating parts of the control signal, we found that more dimensions contribute to the task performance than suggested by traditional synergy analysis. Overall, our work illustrates the emerging possibilities at the interface of muscu-loskeletal physics engines, reinforcement learning and neuro-science to advance our understanding of biological motor control.

neuroscience↗

bioRxiv2023

WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models

Wildlife observation with camera traps has great potential for ethology and ecology, as it gathers data non-invasively in an automated way. However, camera traps produce large amounts of uncurated data, which is time-consuming to annotate. Existing methods to label these data automatically commonly use a fixed pre-defined set of distinctive classes and require many labeled examples per class to be trained. Moreover, the attributes of interest are sometimes rare and difficult to find in large data collections. Large pretrained vision-language models, such as Contrastive Language Image Pretraining (CLIP), offer great promises to facilitate the annotation process of camera-trap data. Images can be described with greater detail, the set of classes is not fixed and can be extensible on demand and pretrained models can help to retrieve rare samples. In this work, we explore the potential of CLIP to retrieve images according to environmental and ecological attributes. We create WildCLIP by fine-tuning CLIP on wildlife camera-trap images and to further increase its flexibility, we add an adapter module to better expand to novel attributes in a few-shot manner. We quantify WildCLIPs performance and show that it can retrieve novel attributes in the Snapshot Serengeti dataset. Our findings outline new opportunities to facilitate annotation processes with complex and multi-attribute captions. The code will be made available at https://github.com/amathislab/wildclip.

ecology↗

bioRxiv2023

Task-driven neural network models predict neural dynamics of proprioception

Proprioception tells the brain the state of the body based on distributed sensors in the body. However, the principles that govern proprioceptive processing from those distributed sensors are poorly understood. Here, we employ a task-driven neural network modeling approach to investigate the neural code of proprioceptive neurons in both cuneate nucleus (CN) and somatosensory cortex area 2 (S1). We simulated muscle spindle signals through musculoskeletal modeling and generated a large-scale, naturalistic movement repertoire to train thousands of neural network models on 16 behavioral tasks, each reflecting a hypothesis about the neural computations of the ascending proprioceptive pathway. We found that the networks internal representations developed through task-optimization generalize from synthetic data to predict single-trial neural activity in CN and S1 of primates performing center-out reaching. Task-driven models outperform linear encoding models and data-driven models. Behavioral tasks, which aim to predict the limb position and velocity were the best to predict the neural activity in both areas. Architectures that are better at solving the tasks are also better at predicting the neural data. Last, since task-optimization develops representations that better predict neural activity during active but not passively generated movements, we hypothesize that neural activity in CN and S1 is top-down modulated during goal-directed movements.

neuroscience↗

bioRxiv2023

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms have revolutionized the analysis of animal behavior, enabling investigators to flexibly quantify behavioral dynamics from conventional video recordings obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into the modules out of which behavior is organized. This challenge is particularly acute because keypoint data is susceptible to high frequency jitter that clustering algorithms can mistake for transitions between behavioral modules. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ("syllables") from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to effectively identify syllables whose boundaries correspond to natural sub-second discontinuities inherent to mouse behavior. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior, and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq therefore renders behavioral syllables and grammar accessible to the many researchers who use standard video to capture animal behavior.

neuroscience↗

bioRxiv2021

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals exhibit diverse behavioral responses, such as exploration and avoidance, to novel cues in the environment. However, it remains unclear how dopamine neuron-related novelty responses influence behavior. Here, we characterized dynamics of novelty exploration using multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq). Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability were explained by a novel reinforcement learning (RL) model of threat prediction, in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus"), and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems. HighlightsO_LINovelty-induced behaviors are analyzed using modern machine-learning methods C_LIO_LINovelty induces risk assessment which develops into engagement or avoidance C_LIO_LIDopamine in the tail of striatum correlates with individual behavioral variability C_LIO_LIReinforcement learning with shaping bonus and uncertainty explains the data C_LI

neuroscience↗

bioRxiv2021

Multi-animal pose estimation and tracking with DeepLabCut

Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having extremely similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, a popular open source pose estimation toolbox, and provide high-performance animal assembly and tracking--features required for robust multi-animal scenarios. Furthermore, we integrate the ability to predict an animals identity directly to assist tracking (in case of occlusions). We illustrate the power of this framework with four datasets varying in complexity, which we release to serve as a benchmark for future algorithm development.

animal behavior and cognition↗

bioRxiv2020

Real-time, low-latency closed-loop feedback using markerless posture tracking

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called DLC-Live! GUI), and integration into (2) Bonsai and (3) AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

neuroscience↗

bioRxiv2020

Task-driven hierarchical deep neural networkmodels of the proprioceptive pathway

Biological motor control is versatile and efficient. Muscles are flexible and undergo continuous changes, requiring distributed adaptive control mechanisms. How proprioception solves this problem in the brain is unknown. The canonical role of proprioception is representing the body state, yet we hypothesize that the proprioceptive system can decode high-level, multi-feature actions. To test this theory, we pursue a task-driven modeling approach.We generated a large synthetic dataset of human arm trajectories tracing the alphabet in 3D space and use a musculoskeletal model plus modeled muscle spindle inputs to extract muscle activity. We then contrast two tasks, one character trajectory-decoding and another action recognition task that allows training of hierarchical models to decode position, or classify the character identity from the spindle firing patterns. Artificial neural networks could robustly solve these tasks, and the networks units show tuning properties akin to neurons in the primate somatosensory cortex and the brainstem. Remarkably, only the action-recognition trained, and not the trajectory decoding trained, models possess directional selective units (which are also uniformly distributed), as in the primate brain. Taken together, our model is the first to link tuning properties in the proprioceptive system at multiple levels to the behavioral level. We find that action-recognition, rather than the canonical trajectory-decoding hypothesis, better explains what is known about the proprioceptive system.

neuroscience↗

bioRxiv2018

Using DeepLabCut for 3D markerless pose estimation across species and behaviors

Noninvasive behavioral tracking of animals during experiments is crucial to many scientific pursuits. Extracting the poses of animals without using markers is often essential for measuring behavioral effects in biomechanics, genetics, ethology & neuroscience. Yet, extracting detailed poses without markers in dynamically changing backgrounds has been challenging. We recently introduced an open source toolbox called DeepLabCut that builds on a state-of-the-art human pose estimation algorithm to allow a user to train a deep neural network using limited training data to precisely track user-defined features that matches human labeling accuracy. Here, with this paper we provide an updated toolbox that is self contained within a Python package that includes new features such as graphical user interfaces and active-learning based network refinement. Lastly, we provide a step-by-step guide for using DeepLabCut.

neuroscience↗

bioRxiv2018

On the inference speed and video-compression robustness of DeepLabCut

Pose estimation is crucial for many applications in neuroscience, biomechanics, genetics and beyond. We recently presented a highly efficient method for markerless pose estimation based on transfer learning with deep neural networks called DeepLabCut. Current experiments produce vast amounts of video data, which pose challenges for both storage and analysis. Here we improve the inference speed of DeepLabCut by up to tenfold and benchmark these updates on various CPUs and GPUs. In particular, depending on the frame size, poses can be inferred offline at up to 1200 frames per second (FPS). For instance, 278 x 278 images can be processed at 225 FPS on a GTX 1080 Ti graphics card. Furthermore, we show that DeepLabCut is highly robust to standard video compression (ffmpeg). Compression rates of greater than 1,000 only decrease accuracy by about half a pixel (for 640 x 480 frame size). DeepLabCuts speed and robustness to compression can save both time and hardware expenses.

animal behavior and cognition↗