Plucking a string or playing a G? Degree of choice abstraction impacts human reinforcement learning

Abstract

In reinforcement learning (RL) experiments, participants learn to make rewarding choices in response to different stimuli; RL models use outcomes to estimate stimulus-response values which change incrementally. RL models consider any response type indiscriminately, ranging from more concretely defined motor choices (e.g. pressing a key with the index finger), to more general choices that can be executed in a number of ways (e.g. getting dinner at the restaurant). But does the learning process vary as a function of the choice type? In Experiment 1, we show that it does: participants were slower and less accurate in learning correct choices of a general format compared to learning more concrete, motor actions. Using computational modeling, we show that two mechanisms contribute to this. First, there was evidence of irrelevant credit assignment: the values of motor actions interfered with the values of other choice dimensions, resulting in more incorrect choices when the correct response is not defined by a single motor action; second, information integration for relevant general choices was slower. In Experiment 2, we replicated and further extended the findings from Experiment 1, by showing that slowed learning was attributable to weaker working memory use, rather than slowed RL learning. In both experiments we ruled out the explanation that the difference in performance between two condition types was driven by difficulty/different levels of complexity. We conclude that defining a more abstract choice space used by multiple learning systems for credit assignment recruits executive resources, limiting how much such processes then contribute to fast learning.