Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations

Jae Young Lee, Jin Bae Park, Yoon Ho Choi

Research output: Contribution to journalArticle

44 Citations (Scopus)

Abstract

This paper focuses on a class of reinforcement learning (RL) algorithms, named integral RL (I-RL), that solve continuous-time (CT) nonlinear optimal control problems with input-affine system dynamics. First, we extend the concepts of exploration, integral temporal difference, and invariant admissibility to the target CT nonlinear system that is governed by a control policy plus a probing signal called an exploration. Then, we show input-to-state stability (ISS) and invariant admissibility of the closed-loop systems with the policies generated by integral policy iteration (I-PI) or invariantly admissible PI (IA-PI) method. Based on these, three online I-RL algorithms named explorized I-PI and integral $Q$ -learning I, II are proposed, all of which generate the same convergent sequences as I-PI and IA-PI under the required excitation condition on the exploration. All the proposed methods are partially or completely model free, and can simultaneously explore the state space in a stable manner during the online learning processes. ISS, invariant admissibility, and convergence properties of the proposed methods are also investigated, and related with these, we show the design principles of the exploration for safe learning. Neural-network-based implementation methods for the proposed schemes are also presented in this paper. Finally, several numerical simulations are carried out to verify the effectiveness of the proposed methods.

Original languageEnglish
Article number6882245
Pages (from-to)916-932
Number of pages17
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume26
Issue number5
DOIs
Publication statusPublished - 2015 May 1

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations'. Together they form a unique fingerprint.

  • Cite this