Biology subjects

Ji-An, L.

Publications and source records attributed to Ji-An, L..

5 recordsLinked to original sources

bioRxiv2025

Human Strategy Adaptation in Reinforcement Learning Resembles Policy Gradient Ascent

A hallmark of intelligence is the ability to adapt behavior to changing environments, which requires adapting ones own learning strategies. This phenomenon is known as learning to learn or meta-learning. Although well established in humans and animals, a computational framework that characterizes how biological agents adapt their learning strategies through experience remains elusive. Here we posit that humans update their learning strategies online through a gradient-based meta-learning process, effectively optimizing how they learn. However, estimating how these strategies evolve over time remains a significant challenge since traditional cognitive models, such as reinforcement learning (RL), typically assume that agents use static strategies. To address this, we introduce DynamicRL, a method that leverages neural networks to estimate the evolution of an individuals RL strategy by tracking cognitive parameters such as learning rates over time. Across four human bandit tasks, DynamicRL consistently outperforms traditional RL models with fixed parameters in fitting behavior, confirming that humans adapt their RL strategies over time. RL parameters estimated by DynamicRL reveal trajectories that systematically increase the expected reward of the RL strategy. The parameter updates at each step resemble policy gradient ascent, and their optimality correlates with the strength of the gradient signal. Moreover, these RL parameters evolve more slowly than decision variables, supporting the hierarchical relationship between strategy learning and value learning. Our work provides a computational framework that expands the hypothesis space from understanding strategies to understanding strategy adaptation, bridging adaptive behavior in biological and artificial intelligence through meta-learning.

animal behavior and cognition↗

bioRxiv2025

Deep Learning Improves Parameter Estimation in Reinforcement Learning Models

AO_SCPLOWBSTRACTC_SCPLOWCognitive models are widely used in psychology and neuroscience to formulate and test hypotheses about cognitive processes. These processes are characterized by model parameters, which are then used for scientific inference. The reliability of scientific conclusions from cognitive modeling depends critically on the reliability of parameter estimation, yet estimating parameters remains a universal challenge particularly when data are too limited to constrain them. In such cases, multiple sets of parameters may explain the experimental data equally well within the same model, raising the question of which parameters are scientifically meaningful. We refer to this problem as parameter ambiguity. In this paper, we investigate parameter ambiguity in reinforcement learning under two optimization methods. We employ the de facto Nelder-Mead method (fminsearch) and a neural network trained to estimate parameters using a modern deep learning pipeline, which has seen limited application in cognitive modeling. Across ten decision-making datasets, we consistently find that the two methods produce substantially different parameter estimates despite achieving nearly identical fitting performance. To address this ambiguity, we introduce a systematic evaluation framework that goes beyond predictive accuracy to assess generalizability, robustness, identifiability, and test-retest reliability, thereby offering principled guidance on which parameter estimates should inform scientific inference. Applying this framework reveals that the neural network with a deep learning pipeline outperforms across these metrics. Our study establishes parameter ambiguity as an underappreciated challenge with significant implications for scientific replicability, highlighting that the choice of optimization method is a critical factor shaping scientific conclusions. We advocate for our multi-faceted evaluation approach to ensure reliable scientific inference and for broader integration of modern deep learning pipelines into cognitive modeling.

animal behavior and cognition↗

bioRxiv2023

Automatic Discovery of Cognitive Strategies with Tiny Recurrent Neural Networks

Normative modeling frameworks such as Bayesian inference and reinforcement learning provide valuable insights into the fundamental principles governing adaptive behavior. While these frameworks are valued for their simplicity and interpretability, their reliance on few parameters often limits their ability to capture realistic biological behavior, leading to cycles of handcrafted adjustments that are prone to research subjectivity. Here, we present a novel modeling approach leveraging recurrent neural networks to discover the cognitive algorithms governing biological decision-making. We show that neural networks with just 1-4 units often outperform classical cognitive models and match larger neural networks in predicting the choices of individual animals and humans across six well-studied reward learning tasks. Critically, we then interpret the trained networks using dynamical systems concepts, enabling a unified comparison of cognitive models and revealing detailed mechanisms underlying choice behavior. Our approach also estimates the dimensionality of behavior and offers insights into algorithms implemented by AI agents trained in a meta-reinforcement learning setting. Overall, we present a systematic approach for discovering interpretable cognitive strategies in decision-making, offering insights into neural mechanisms and a foundation for studying both healthy and dysfunctional cognition.

animal behavior and cognition↗

bioRxiv2019

Face familiarity detection with complex synapses

Synaptic plasticity is a complex phenomenon involving multiple biochemical processes that operate on different timescales. We recently showed that this complexity can greatly increase the memory capacity of neural networks when the variables that characterize the synaptic dynamics have limited precision, as in biological systems. These types of complex synapses have been tested mostly on simple memory retrieval problems involving random and uncorrelated patterns. Here we turn to a real-world problem, face familiarity detection, and we show that also in this case it is possible to take advantage of synaptic complexity to store in memory a large number of faces that can be recognized at a later time. In particular, we show that the familiarity memory capacity of a system with complex synapses grows almost linearly with the number of the synapses and quadratically with the number of neurons. Complex synapses are superior to simple ones, which are characterized by a single variable, even when the total number of dynamical variables is matched. We further show that complex and simple synapses have distinct signatures that are testable in proposed experiments. Our results indicate that a memory system with complex synapses can be used in real-world tasks such as face familiarity detection. SignificanceThe complexity of biological synapses is probably important for enabling us to remember the past for a long time and rapidly store new memories. The advantage of complex synapses in terms of memory capacity is significant when the variables that characterize the synaptic dynamics have limited precision. This advantage has been estimated under the simplifying assumption that the memories to be stored are random and uncorrelated. Here we show that synaptic complexity is important also in a more challenging and realistic face familiarity detection task. We built a simple neural circuit that can report whether a face has been previously seen or not. This circuit incorporates complex synapses that operate on multiple timescales. The memory performance of this circuit is significantly higher than in the case in which synapses are simple, indicating that the complexity of biological synapses can be important also in real-world memory tasks.

neuroscience↗

bioRxiv2019

Understanding the functional and structural differences across excitatory and inhibitory neurons

One of the most fundamental organizational principles of the brain is the separation of excitatory (E) and inhibitory (I) neurons. In addition to their opposing effects on post-synaptic neurons, E and I cells tend to differ in their selectivity and connectivity. Although many such differences have been characterized experimentally, it is not clear why they exist in the first place. We studied this question in an artificial neural network equipped with multiple E and I cell types. We found that a deep convolutional recurrent network trained to perform an object classification task was able to capture salient distinctions between E and I neurons. We explored the necessary conditions for the network to develop distinct selectivity and connectivity across cell types. We found that neurons that project to higher-order areas will have greater stimulus selectivity, regardless of whether they are excitatory or not. Sparser connectivity is required for higher selectivity, but only when the recurrent connections are excitatory. These findings demonstrate that the differences observed across E and I neurons are not independent, and can be explained using a smaller number of factors.

neuroscience↗