Research Article
Distributed Reinforcement Learning with States Feature Encoding and States Stacking in Continuous Action Space
@INPROCEEDINGS{10.1007/978-3-030-67537-0_21, author={Tianqi Xu and Dianxi Shi and Zhiyuan Wang and Xucan Chen and Yaowen Zhang}, title={Distributed Reinforcement Learning with States Feature Encoding and States Stacking in Continuous Action Space}, proceedings={Collaborative Computing: Networking, Applications and Worksharing. 16th EAI International Conference, CollaborateCom 2020, Shanghai, China, October 16--18, 2020, Proceedings, Part I}, proceedings_a={COLLABORATECOM}, year={2021}, month={1}, keywords={Distributed reinforcement learning Importance sampling Continuous control task Scalable agent}, doi={10.1007/978-3-030-67537-0_21} }- Tianqi Xu
Dianxi Shi
Zhiyuan Wang
Xucan Chen
Yaowen Zhang
Year: 2021
Distributed Reinforcement Learning with States Feature Encoding and States Stacking in Continuous Action Space
COLLABORATECOM
Springer
DOI: 10.1007/978-3-030-67537-0_21
Abstract
The practical application of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. This work, we utilize the distributed reinforcement learning architecture to deal with continuous control tasks. The Importance Weighted Actor-Learner Architectures (IMPALA) decouples the acting and learning process to reduce queuing time. IMPALA attains higher scores on the new DMLab-30 set and the Atari-57 set because of its high performance, good scalability, and high efficiency. We extend IMPALA on the continuous control tasks with three changes. We encoder states into low dimensional data to establish an action distribution function that the agents have the ability to exploit and explore. A queue buffer is used to store a mini-batch data and discard them after training. In order to make the agent take appropriate action in the continuous control environment, we stack the past three steps states that attempt to make the robot moves smoothly. Finally, experiments are carried out on Mujoco tasks. The results show that our work is better than other distributed reinforcement learning algorithms.
