Multiagent systems are rapidly finding applications in a variety of domains, including robotics, distributed control, telecommunications, and economics. The complexity of many tasks arising in these domains makes them difficult to solve with preprogrammed agent behaviors. The agents must, instead, discover a solution on their own, using learning. A significant part of the research on multiagent learning concerns reinforcement learning techniques. This paper provides a comprehensive survey of multiagent reinforcement learning (MARL). A central issue in the field is the formal statement of the multiagent learning goal. Different viewpoints on this issue have led to the proposal of many different goals, among which two focal points can be distinguished: stability of the agents' learning dynamics, and adaptation to the changing behavior of the other agents. The MARL algorithms described in the literature aim---either explicitly or implicitly---at one of these two goals or at a combination of both, in a fully cooperative, fully competitive, or more general setting. A representative selection of these algorithms is discussed in detail in this paper, together with the specific issues that arise in each category. Additionally, the benefits and challenges of MARL are described along with some of the problem domains where the MARL techniques have been applied. Finally, an outlook for the field is provided.

/pdf/a-comprehensive-survey-of-multiagent-reinforcement-learning-1mwdfqzdug.pdf

A Comprehensive Survey of Multiagent Reinforcement Learning

Wheeled vehicles are often incapable of transporting heavy materials over rough terrain or up staircases. Lower extremity exoskeletons supplement human intelligence with the strength and endurance of a pair of wearable robotic legs that support a payload. This paper summarizes the design and analysis of the Berkeley lower extremity exoskeleton (BLEEX). The anthropomorphically based BLEEX has 7 DOF per leg, four of which are powered by linear hydraulic actuators. The selection of the DOF, critical hardware design aspects, and initial performance measurements of BLEEX are discussed.

Biomechanical design of the Berkeley lower extremity exoskeleton (BLEEX)

The Quark and the Jaguar

Considerable scientific and technological efforts have been devoted to develop neuroprostheses and hybrid bionic systems that link the human nervous system with electronic or robotic prostheses, with the main aim of restoring motor and sensory functions in disabled patients. A number of neuroprostheses use interfaces with peripheral nerves or muscles for neuromuscular stimulation and signal recording. Herein, we provide a critical overview of the peripheral interfaces available and trace their use from research to clinical application in controlling artificial and robotic prostheses. The first section reviews the different types of non-invasive and invasive electrodes, which include surface and muscular electrodes that can record EMG signals from and stimulate the underlying or implanted muscles. Extraneural electrodes, such as cuff and epineurial electrodes, provide simultaneous interface with many axons in the nerve, whereas intrafascicular, penetrating, and regenerative electrodes may contact small groups of axons within a nerve fascicle. Biological, technological, and material science issues are also reviewed relative to the problems of electrode design and tissue injury. The last section reviews different strate- gies for the use of information recorded from peripheral interfaces and the current state of control neuroprostheses and hybrid bionic systems.

A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems

Metaheuristics are the most exciting development in approximate optimization techniques of the last two decades. They have had widespread successes in attacking a variety of difficult combinatorial optimization problems that arise in many practical areas. This bibliography provides a classification of a comprehensive list of 1380 references on the theory and application of metaheuristics. Metaheuristics include but are not limited to constraint logic programming; greedy random adaptive search procedures; natural evolutionary computation; neural networks; non-monotonic search strategies; space-search methods; simulated annealing; tabu search; threshold algorithms and their hybrids. References are presented in alphabetical order under a number of subheadings.

Metaheuristics: A bibliography

This paper reports the prosthetic hand controller discriminating ten forearm motions from two channels of EMG signals. The controller uses the real-time learning method that is defined as simultaneously controlling a prosthetic hand and learning to adapt to the operator's characteristics. In this method, the controller is divided into three units. The analysis unit extracts useful information for discriminating motions from EMG. The adaptation unit learns the relation between EMG and control command and adapts to the operator's characteristics. The trainer unit generates training data and makes the adaptation unit learn in real-time. In experiments, the proposed controller performs discriminating a maximum of ten forearm motions including four wrist motions and six hand motions. In an eight forearm motions experiment, the five subjects' average discriminating rate, which serves an index of accurate controlling, was 85.1%. Two groups occur from this result, one marks a high performance (91.7%) and another does not (75.2%). The paper discusses the factors of this difference in performance from both phases of training and reasons that the low proficiency leads to undesirable results in the latter groups. Besides, in the ten forearm motions experiment the average discriminating rate of three subjects who achieve high performance in the previous experiment was 91.5%. This paper concludes that the effectiveness of the real-time learning method is confirmed by these experiments.

EMG prosthetic hand controller using real-time learning method

An approach to the pursuit problem on a heterogeneous multiagent system using reinforcement learning

This paper proposes a novel learning method for prosthetic hand control. Conventional works have used off-line learning methods for control, and hence two kinds of training must be carried out separately: one is for the amputee to control prosthetic hands, and the other is for the prosthetic hand controller to adapt to the amputee's variations. Consequently, an amputee cannot acquire the sensations of operating prosthetic hands through training, and nevertheless he or she is likely to experience difficulties in forcing the prosthetic hand controller to follow the change of his or her own characteristics in practical use. We accordingly design an on-line learning mechanism which can adapt to the individual's characteristics in real time. Using this mechanism, a suitable mapping function of the surface electromyogram (EMG) to motions of prosthetic hands can be acquired according to the amputee's evaluation in practical use. Thereby, the mechanism realizes a shortening of training time and adaptation to individual variation in real time. The experiments succeeded in discriminating six forearm motions to verify the proposed method. First, we use intrinsic exercise images to control a prosthetic hand, and compare our on-line method with one conventional off-line method. Second, we use EMG signals on shoulder girdles to control the prosthetic hand for upper elbow amputation. The discrimination rate in forearm EMG experiments is 89.9% by our method and 80.3% by the conventional method. Moreover, we show the possibility of applying the on-line learning method to upper elbow amputees, because a discrimination rate of 79.3% is achieved by our method in shoulder girdle EMG classification. © 2001 Scripta Technica, Electron Comm Jpn Pt 3, 84(10): 35–46, 2001

On-line learning method for EMG prosthetic hand control

We discuss the necessity of a learning mechanism for an EMG prosthetic hand controller, and the real-time learning method is proposed and designed. This method divides the controller into three units. The analysis unit extracts useful informations for discriminating motions from the EMG. The adaptation unit learns the relation between EMG and control command and adapts operator's characteristics. The trainer unit makes the adaptation unit learn in real-time. Experiments show that the proposed controller discriminates ten forearm motions, which contain four wrist motions and six hand motions, and learns within 4/spl sim/25 minutes. The average of the discriminating rate is 91.5%.

EMG prosthetic hand controller discriminating ten motions using real-time learning method

In this paper, we propose a power assist device for lower back flexion and extension, when carrying a heavy load. To see the effect of the device, we model a human body and analyze a compression force in his lower back, as well as the evaluation of a supported force at a hand position. A prototype of the device is manufactured and a controller of the device is developed, which can follow a voluntary human motion assisting his strength.

Yukinori Kakazu

Papers

EMG prosthetic hand controller using real-time learning method

An approach to the pursuit problem on a heterogeneous multiagent system using reinforcement learning

On-line learning method for EMG prosthetic hand control

EMG prosthetic hand controller discriminating ten motions using real-time learning method

Development of wearable exoskeleton power assist system for lower back support