Fictitious play based Markov game control for robotic arm manipulator
01 Oct 2014-pp 1-6
TL;DR: The work attempts a `safe yet consistent' Markov game controller which advocates a minimax policy during the startup control stages and later moves to a more enterprising policy based on stochastic fictitious play.
Abstract: Markov games can be used as a platform to deal with exogenous disturbances and parametric variations. In this work an attempt has been made to achieve a superior performance with fuzzy Markov game based control by hybridizing two game theory based approaches of ‘fictitious play’ and ‘minimax’. The work attempts a ‘safe yet consistent’ Markov game controller which advocates a minimax policy during the startup control stages and later moves to a more enterprising policy based on stochastic fictitious play. The proposed controller addresses continuous state action space problems wherein we use fuzzy inference system as a universal approximator for generalization with a proportional derivative control in the nested position tracking loop. The proposed controller is simulated on a two link robot and its performance compared against fuzzy Markov game control and fuzzy Q control. Simulation results elucidate the fact that proposed control scheme leads to an improved controller with lower tracking error and torque requirements.
Citations
More filters
TL;DR: A Nash-based feedback control law is formulated for an Euler–Lagrange system to yield a solution to noncooperative differential game.
Abstract: We formulate a Nash-based feedback control law for an Euler–Lagrange system to yield a solution to noncooperative differential game. The robot manipulators are broadly used in industrial units on t...
3 citations
Cites background from "Fictitious play based Markov game c..."
...Sharma and Gopal (2014) tried to achieve a superior performance with fuzzy Markov game-based control by hybridizing two game theory–based approaches of “fictitious play” and “minimax.”...
[...]
22 Feb 2020
TL;DR: In this paper, a novel variant of fictitious play is proposed, by considering multi-model adaptive filters as a method to estimate other players' strategies, which can be used as a coordination mechanism between players when they should take decisions under uncertainty.
Abstract: This paper casts coordination of a team of robots within the framework of game theoretic learning algorithms. In particular a novel variant of fictitious play is proposed, by considering multi-model adaptive filters as a method to estimate other players’ strategies. The proposed algorithm can be used as a coordination mechanism between players when they should take decisions under uncertainty. Each player chooses an action after taking into account the actions of the other players and also the uncertainty. Uncertainty can occur either in terms of noisy observations or various types of other players. In addition, in contrast to other game-theoretic and heuristic algorithms for distributed optimisation, it is not necessary to find the optimal parameters a priori. Various parameter values can be used initially as inputs to different models. Therefore, the resulting decisions will be aggregate results of all the parameter values. Simulations are used to test the performance of the proposed methodology against other game-theoretic learning algorithms.
References
More filters
TL;DR: Simulations of the proposed reinforcement adaptive fuzzy control scheme on the cart-pole balancing problem and a two-degree-of freedom (2DOF) manipulator, SCARA robot arm verify the effectiveness of the approach.
70 citations
TL;DR: The proposed Markov game-adaptive fuzzy controller outperformed other controllers in terms of tracking errors and control torque requirements, over different desired trajectories, and demonstrates the viability of FISs for accelerating learning in Markov games and extending Markovgame-based control to continuous state-action space problems.
Abstract: This paper develops an adaptive fuzzy controller for robot manipulators using a Markov game formulation. The Markov game framework offers a promising platform for robust control of robot manipulators in the presence of bounded external disturbances and unknown parameter variations. We propose fuzzy Markov games as an adaptation of fuzzy Q-learning (FQL) to a continuous-action variation of Markov games, wherein the reinforcement signal is used to tune online the conclusion part of a fuzzy Markov game controller. The proposed Markov game-adaptive fuzzy controller uses a simple fuzzy inference system (FIS), is computationally efficient, generates a swift control, and requires no exact dynamics of the robot system. To illustrate the superiority of Markov game-adaptive fuzzy control, we compare the performance of the controller against a) the Markov game-based robust neural controller, b) the reinforcement learning (RL)-adaptive fuzzy controller, c) the FQL controller, d) the Hinfin theory-based robust neural game controller, and e) a standard RL-based robust neural controller, on two highly nonlinear robot arm control problems of i) a standard two-link rigid robot arm and ii) a 2-DOF SCARA robot manipulator. The proposed Markov game-adaptive fuzzy controller outperformed other controllers in terms of tracking errors and control torque requirements, over different desired trajectories. The results also demonstrate the viability of FISs for accelerating learning in Markov games and extending Markov game-based control to continuous state-action space problems.
39 citations
TL;DR: In this article, a comparative study of two different adaptive control algorithms, Lyapunov stability-based approach and neural adaptive control, is presented for trajectory tracking control of a two-degree-of-freedom (2DOF) manipulator.
Abstract: A comparative study evaluates the problem of determining the control that must be exerted on manipulator joints. Two different techniques are studied: (i) direct and indirect adaptive controls and (ii) neural adaptive control. In the direct adaptive technique the Lyapunov stability-based approach is used with the objective of minimizing the tracking errors of the joints in the adaptation process. In the indirect adaptive technique the regulator parameters are updated via the estimation of the process model. This step, using a recursive least squares algorithm, is based on the error at the input and on the filtered dynamic model in order to avoid acceleration measurements.
Neural adaptive control is based on learning from input-output measurements and not on parametricmodel-based dynamics. It is important to note that adaptive control requires a real-time estimation of the system parameters and a well-defined dynamic model, whereas neural adaptive control does not require any of these conditions.
All the above-mentioned techniques are applied to the trajectory-tracking control of a two-degree-of-freedom (2DOF) manipulator. the experimental results show the effectiveness of the neural adaptive techniques for the trajectory-tracking errors.
24 citations
01 Jun 2007
TL;DR: For the robot control task, the proposed controller achieved superior robustness to changes in payload mass and external disturbances, over other control schemes, and validate the effectiveness of neural networks in extending the Markov game framework to problems with continuous state-action spaces.
Abstract: This paper proposes a reinforcement learning (RL)-based game-theoretic formulation for designing robust controllers for nonlinear systems affected by bounded external disturbances and parametric uncertainties. Based on the theory of Markov games, we consider a differential game in which a 'disturbing' agent tries to make worst possible disturbance while a 'control' agent tries to make best control input. The problem is formulated as finding a min-max solution of a value function. We propose an online procedure for learning optimal value function and for calculating a robust control policy. Proposed game-theoretic paradigm has been tested on the control task of a highly nonlinear two-link robot system. We compare the performance of proposed Markov game controller with a standard RL-based robust controller, and an H"~ theory-based robust game controller. For the robot control task, the proposed controller achieved superior robustness to changes in payload mass and external disturbances, over other control schemes. Results also validate the effectiveness of neural networks in extending the Markov game framework to problems with continuous state-action spaces.
16 citations