High Path Converging Reward Space: Imitating Human Behavior in Robot Arm Control

In this paper, we suggest High Path Converging Reward Space (HPCR) as a reinforcement learning reward shaping method. HPCR helps robot arms learn quickly and stably in certain environments that require specific actions and are difficult to receive rewards. In environments where a person reaches a goal at a distance, he or she tends to reach the goal by first determining the direction to throw and then increasing the speed or strength. To imitate this way of human behavior, the initial reward space is defined as the space between the robot arm and the goal point. It allows the robot arm to learn along the path in that direction. Then, when the robot arm proceeds with a certain number of successes or episodes, the reward space range gradually converges toward the goal point, eventually becoming the size of the actual goal size. In addition, HPCR can minimize the range in the z-axis direction of the reward space by considering only the value at the maximum height of the goal, even if the environment is complicated. Using HPCR with SAC+HER and TQC+HER algorithm, it is possible to stably reach a position that cannot be reached using the existing reward space.