Computational characteristics of the striatal dopamine system described by reinforcement learning with fast generalization

Abstract

Generalization enables applying past experience to similar but nonidentical situations. Therefore, it may be essential for adaptive behaviors. Recent neurobiological observation indicates that the striatal dopamine system achieves generalization and subsequent discrimination by updating corticostriatal synaptic connections in differential response to reward and punishment. To analyze how the computational characteristics in this system affect behaviors, we proposed a novel reinforcement learning model with multilayer neural networks in which the synaptic weights of only the last layer are updated according to the prediction error. We set fixed connections between the input and hidden layers so as to maintain the similarity of inputs in the hidden-layer representation. This network enabled fast generalization, and thereby facilitated safe and efficient exploration in reinforcement learning tasks, compared to algorithms which do not show generalization. However, disturbance in the network induced aberrant valuation. In conclusion, the unique computation suggested by corticostriatal plasticity has the advantage of providing safe and quick adaptations to unknown environments, but on the other hand has the potential defect which can induce maladaptive behaviors like delusional symptoms of psychiatric disorders. Author summaryThe brain has an ability to generalize knowledge obtained from reward- and punishment-related learning. Animals that have been trained to associate a stimulus with subsequent reward or punishment respond not only to the same stimulus but also to resembling stimuli. How does generalization affect behaviors in situations where individuals are required to adapt to unknown environments? It may enable efficient learning and promote adaptive behaviors, but inappropriate generalization may disrupt behaviors by associating reward or punishment with irrelevant stimuli. The effect of generalization here should depend on computational characteristics of underlying biological basis in the brain, namely, the striatal dopamine system. In this research, we made a novel computational model based on the characteristics of the striatal dopamine system. Our model enabled fast generalization and showed its advantage of providing safe and quick adaptation to unknown environments. By contrast, disturbance of our model induced abnormal behaviors. The results suggested the advantage and the shortcoming of generalization by the striatal dopamine system.