Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

Co-Planar Stereotaxic Atlas of the Human Brain—3-Dimensional Proportional System: An Approach to Cerebral Imaging, J. Talairach, P. Tournoux. Georg Thieme Verlag, New York (1988), 122 pp., 130 figs. DM 268

I have developed "tennis elbow" from lugging this book around the past four weeks, but it is worth the pain, the effort, and the aspirin. It is also worth the (relatively speaking) bargain price. Including appendixes, this book contains 894 pages of text. The entire panorama of the neural sciences is surveyed and examined, and it is comprehensive in its scope, from genomes to social behaviors. The editors explicitly state that the book is designed as "an introductory text for students of biology, behavior, and medicine," but it is hard to imagine any audience, interested in any fragment of neuroscience at any level of sophistication, that would not enjoy this book. The editors have done a masterful job of weaving together the biologic, the behavioral, and the clinical sciences into a single tapestry in which everyone from the molecular biologist to the practicing psychiatrist can find and appreciate his or

Principles of Neural Science

The hypothesis that dopamine is important for reward has been proposed in a number of forms, each of which has been challenged. Normally, rewarding stimuli such as food, water, lateral hypothalamic brain stimulation and several drugs of abuse become ineffective as rewards in animals given performance-sparing doses of dopamine antagonists. Dopamine release in the nucleus accumbens has been linked to the efficacy of these unconditioned rewards, but dopamine release in a broader range of structures is implicated in the 'stamping-in' of memory that attaches motivational importance to otherwise neutral environmental stimuli.

Dopamine, learning and motivation

http://www.columbia.edu/itc/gsas/g9600/2003/JavitchReadings/Schultz(Sulzer).pdf

Getting Formal with Dopamine and Reward

Objective Electroencephalography (EEG) analysis has been an important tool in neuroscience with applications in neuroscience, neural engineering (e.g. Brain-computer interfaces, BCI's), and even commercial applications. Many of the analytical tools used in EEG studies have used machine learning to uncover relevant information for neural classification and neuroimaging. Recently, the availability of large EEG data sets and advances in machine learning have both led to the deployment of deep learning architectures, especially in the analysis of EEG signals and in understanding the information it may contain for brain functionality. The robust automatic classification of these signals is an important step towards making the use of EEG more practical in many applications and less reliant on trained professionals. Towards this goal, a systematic review of the literature on deep learning applications to EEG classification was performed to address the following critical questions: (1) Which EEG classification tasks have been explored with deep learning? (2) What input formulations have been used for training the deep networks? (3) Are there specific deep learning network structures suitable for specific types of tasks? Approach A systematic literature review of EEG classification using deep learning was performed on Web of Science and PubMed databases, resulting in 90 identified studies. Those studies were analyzed based on type of task, EEG preprocessing methods, input type, and deep learning architecture. Main results For EEG classification tasks, convolutional neural networks, recurrent neural networks, deep belief networks outperform stacked auto-encoders and multi-layer perceptron neural networks in classification accuracy. The tasks that used deep learning fell into five general groups: emotion recognition, motor imagery, mental workload, seizure detection, event related potential detection, and sleep scoring. For each type of task, we describe the specific input formulation, major characteristics, and end classifier recommendations found through this review. Significance This review summarizes the current practices and performance outcomes in the use of deep learning for EEG classification. Practical suggestions on the selection of many hyperparameters are provided in the hope that they will promote or guide the deployment of deep learning to EEG datasets in future research.

https://iopscience.iop.org/article/10.1088/1741-2552/ab0ab5/pdf

Deep learning for electroencephalogram (EEG) classification tasks: a review.

Parkinsonism reduces coordination of fingers, wrist, and arm in fine motor control

It is generally thought that the signal-to-noise ratio, the bandwidth, and the information content of neural data acquired via noninvasive scalp electroencephalography (EEG) are insufficient to extract detailed information about natural, multijoint movements of the upper limb. Here, we challenge this assumption by continuously decoding three-dimensional (3D) hand velocity from neural data acquired from the scalp with 55-channel EEG during a 3D center-out reaching task. To preserve ecological validity, five subjects self-initiated reaches and self-selected targets. Eye movements were controlled so they would not confound the interpretation of the results. With only 34 sensors, the correlation between measured and reconstructed velocity profiles compared reasonably well to that reported by studies that decoded hand kinematics from neural activity acquired intracranially. We subsequently examined the individual contributions of EEG sensors to decoding to find substantial involvement of scalp areas over the sensorimotor cortex contralateral to the reaching hand. Using standardized low-resolution brain electromagnetic tomography (sLORETA), we identified distributed current density sources related to hand velocity in the contralateral precentral gyrus, postcentral gyrus, and inferior parietal lobule. Furthermore, we discovered that movement variability negatively correlated with decoding accuracy, a finding to consider during the development of brain-computer interface systems. Overall, the ability to continuously decode 3D hand velocity from EEG during natural, center-out reaching holds promise for the furtherance of noninvasive neuromotor prostheses for movement-impaired individuals.

https://www.jneurosci.org/content/jneuro/30/9/3432.full.pdf

Reconstructing Three-Dimensional Hand Movements from Noninvasive Electroencephalographic Signals

If visual feedback is discordant with movement direction, the visuo-motor mapping is disrupted, but can be updated with practice. In this experiment subjects practiced discrete arm movements under conditions of visual feedback rotation. One group was exposed to 10°-step increments of visual feedback rotation up to a total of 90°, a second group to a 90° visual feedback rotation throughout the experiment. After the first group reached the 90° visual feedback rotation, its subjects performed faster, with less spatial error, and showed larger aftereffects than the subjects who practiced constantly under the 90° visual feedback rotation condition. Results suggest that gradually increasing feedback distortion allows more complete adaptation than a large, sudden distortion onset.

Adaptation to gradual as compared with sudden visuo-motor distortions.

Stroke significantly affects thousands of individuals annually, leading to considerable physical impairment and functional disability. Gait is one of the most important activities of daily living affected in stroke survivors. Recent technological developments in powered robotics exoskeletons can create powerful adjunctive tools for rehabilitation and potentially accelerate functional recovery. Here, we present the development and evaluation of a novel lower limb robotic exoskeleton, namely H2 (Technaid S.L., Spain), for gait rehabilitation in stroke survivors. H2 has six actuated joints and is designed to allow intensive overground gait training. An assistive gait control algorithm was developed to create a force field along a desired trajectory, only applying torque when patients deviate from the prescribed movement pattern. The device was evaluated in 3 hemiparetic stroke patients across 4 weeks of training per individual (approximately 12 sessions). The study was approved by the Institutional Review Board at the University of Houston. The main objective of this initial pre-clinical study was to evaluate the safety and usability of the exoskeleton. A Likert scale was used to measure patient’s perception about the easy of use of the device. Three stroke patients completed the study. The training was well tolerated and no adverse events occurred. Early findings demonstrate that H2 appears to be safe and easy to use in the participants of this study. The overground training environment employed as a means to enhance active patient engagement proved to be challenging and exciting for patients. These results are promising and encourage future rehabilitation training with a larger cohort of patients. The developed exoskeleton enables longitudinal overground training of walking in hemiparetic patients after stroke. The system is robust and safe when applied to assist a stroke patient performing an overground walking task. Such device opens the opportunity to study means to optimize a rehabilitation treatment that can be customized for individuals. Trial registration: This study was registered at ClinicalTrials.gov (
 https://clinicaltrials.gov/show/NCT02114450
 ).

/pdf/the-h2-robotic-exoskeleton-for-gait-rehabilitation-after-1ryhup8jx3.pdf

Jose L. Contreras-Vidal

Papers

Deep learning for electroencephalogram (EEG) classification tasks: a review.

Parkinsonism reduces coordination of fingers, wrist, and arm in fine motor control

Reconstructing Three-Dimensional Hand Movements from Noninvasive Electroencephalographic Signals

Adaptation to gradual as compared with sudden visuo-motor distortions.

The H2 robotic exoskeleton for gait rehabilitation after stroke: early findings from a clinical study