This paper describes a method for implementing a gradient-based nonlinear optimization technique for optimal tuning of nonlinear power system controllers such as a flexible ac transmission system (FACTS) via a combination of hardware and software in the real-time digital simulator (RTDS). The RTDS offers fundamental advantages over other commercial software tools because it is a fully digital power system simulator that operates in real-time. Therefore, nonlinear optimal parameters can be effectively determined under similar environments to actual physical systems by this real-time implementation. In this paper, Broyden-Fletcher-Goldfarb-Shanno (BFGS) optimization algorithm, which has a quadratic convergence property, is implemented in order to design the nonlinear controllers in power systems. Its effectiveness is demonstrated with two design examples intended to improve low-frequency oscillation damping in power systems. After illustrating an example to optimize nonlinear parameters of the power system stabilizer (PSS), the paper introduces optimal design of an external controller (EC) for the series capacitive reactance compensator (SCRC), which is one of the FACTS devices, in real-time simulation.