%A LI,CAI %A Lowe,Robert %A Ziemke,Tom %D 2013 %J Frontiers in Neurorobotics %C %F %G English %K reinforcement learning,cpg-actor-critic,DST,SI,value system %Q %R 10.3389/fnbot.2013.00005 %W %L %M %P %7 %8 2013-April-08 %9 Original Research %+ Mr CAI LI,University of Skövde,IKI,Högskolevägen 1,Skövde,541 28,Gotland,Sweden,gauss.lee@his.se %# %! Learning to Walk based on CPGs by Using Natural-Actor-Critic architecture %* %< %T Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture %U https://www.frontiersin.org/articles/10.3389/fnbot.2013.00005 %V 7 %0 JOURNAL ARTICLE %@ 1662-5218 %X The identification of learning mechanisms for locomotion has been the subject of much research for some time but many challenges remain. Dynamic systems theory (DST) offers a novel approach to humanoid learning through environmental interaction. Reinforcement learning (RL) has offered a promising method to adaptively link the dynamic system to the environment it interacts with via a reward-based value system. In this paper, we propose a model that integrates the above perspectives and applies it to the case of a humanoid (NAO) robot learning to walk the ability of which emerges from its value-based interaction with the environment. In the model, a simplified central pattern generator (CPG) architecture inspired by neuroscientific research and DST is integrated with an actor-critic approach to RL (cpg-actor-critic). In the cpg-actor-critic architecture, least-square-temporal-difference based learning converges to the optimal solution quickly by using natural gradient learning and balancing exploration and exploitation. Futhermore, rather than using a traditional (designer-specified) reward it uses a dynamic value function as a stability indicator that adapts to the environment. The results obtained are analyzed using a novel DST-based embodied cognition approach. Learning to walk, from this perspective, is a process of integrating levels of sensorimotor activity and value.