An Empirical Investigation of Early Stopping Optimizations in Proximal Policy Optimization

IEEE Access

Shengyi Huang¹  Santiago Ontanon¹  Takashi Matsubara²  Rousslan Fernand Julien Dossa³ 

[1] College of Computing & Informatics, Drexel University, Philadelphia, PA, USA;Graduate School of Engineering Science, Osaka University, Osaka, Japan;Graduate School of System Informatics, Kobe University, Hyogo, Japan;

关键词: Artificial Intelligence;    deep learning;    reinforcement learning;    proximal policy optimization;    robotics and automation;    robot learning;   

DOI  :  10.1109/ACCESS.2021.3106662

来源: DOAJ

【 摘 要 】

Code-level optimizations, which are low-level optimization techniques used in the implementation of algorithms, have generally been considered as tangential and often do not appear in published pseudo-code of Reinforcement Learning (RL) algorithms. However, recent studies suggest these optimizations to be critical to the performance of algorithms such as Proximal Policy Optimization (PPO). In this paper, we investigate the effect of one such optimization known as “early stopping” implemented for PPO in the popular openai/spinningup library but not in openai/baselines. This optimization technique, which we refer to as KLE-Stop, can stop the policy update within an epoch if the mean Kullback-Leibler (KL) Divergence between the target policy and current policy becomes too high. More specifically, we conduct experiments to examine the empirical importance of KLE-Stop and its conservative variant KLE-Rollback when they are used in conjunction with other common code-level optimizations. The main findings of our experiments are 1) the performance of PPO is sensitive to the number of update iterations per epoch ( $K$ ), 2) Early stopping optimizations (KLE-Stop and KLE-Rollback) mitigate such sensitivity by dynamically adjusting the actual number of update iterations within an epoch, 3) Early stopping optimizations could serve as a convenient alternative to tuning on $K$ .

【 授权许可】

Unknown