Computational mechanisms of osmoregulation: a reinforcement learning model for sodium appetite

Abstract

Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, and reward learning. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multi-modal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation via sodium appetitive behavior can be understood as a reinforcement learning process and provide a mechanistic explanation for the underlying neural mechanisms of sodium appetite and homeostatic behavior. Author SummaryThe taste of high-concentration saltwater is rewarding during sodium depletion, while it is aversive in a sodium sufficient state. This "sodium appetite" is a clear manifestation of homeostasis maintenance and proper action selection in animals, reflecting the internal environment. To reveal the computational mechanism of this property, we applied a machine learning model, in which homeostatic stability is a reward and the goal is to maximize the sum of the reward, and simulated animal behavioral experiments. The results suggest that the mechanism of sodium-appetite behavior is based on the machine learning model. Specifically, by replicating the results of neural circuit manipulation, which controls sodium appetite, an algorithm in which the function of a neural population affects sodium appetite behaviors was proposed. Our results provide a fundamental computational model for a mechanism by a function of a neural cell type to regulate animal behavior. More generally, this study can be fundamental to understanding the computational process of decision making reflecting the internal environment.