Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia

Frontiers in Neuroscience

Mehdi eKhamassi²  Kevin eGurney³  Mark D Humphries³ 

[1] Ecole Normale Superieure;UMR7222, Centre National de la Recherche Scientifique;University of Sheffield;Université Pierre et Marie Curie;

关键词: Decision Making;    reinforcement learning;    Reward;    uncertainty;    meta-parameters;   

DOI  :  10.3389/fnins.2012.00009

来源: DOAJ

【 摘 要 】

We continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This`exploration-exploitation' trade-off depends on the environment: stability favours exploiting knowledge to maximise gains; volatility favours exploring new options and discovering new outcomes. Here we set out to reconcile recent evidence for dopamine's involvement in the exploration-exploitation trade-off with the existing evidence for basal ganglia control of action selection, by testing the hypothesis that tonic dopamine in the striatum, the basal ganglia's input nucleus, sets the current exploration-exploitation tradeoff. We first advance the idea of interpreting the basal ganglia output as a probability distribution function for action selection. Using computational models of the full basal ganglia circuit, we showed that, under this interpretation, the actions of dopamine within the striatum change the basal ganglia's output to favour the level of exploration or exploitation encoded in the probability distribution. We also found that our models predict striatal dopamine controls the exploration-exploitation trade-off if we instead read out the probability distribution from the target nuclei of the basal ganglia, where their inhibitory input shapes the cortical input to these nuclei. Finally, by integrating the basal ganglia within a reinforcement learning model, we showed how dopamine's effect on the exploration-exploitation trade-off could be measurable in a forced two-choice task. These simulations also showed how tonic dopamine can appear to affect learning while only directly altering the trade-off. Thus, our models support the hypothesis that changes in tonic dopamine within the striatum can alter the exploration-exploitation trade-off by modulating the output of the basal ganglia.

【 授权许可】

Unknown