Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.

/pdf/reinforcement-learning-an-introduction-rzxgej9p17.pdf

Reinforcement Learning: An Introduction

https://is.tuebingen.mpg.de/fileadmin/user_upload/files/publications/Neural-Netw-2008-21-682_4867[0].pdf

2008 Special Issue: Reinforcement learning of motor skills with policy gradients

Policy-gradient-based actor-critic algorithms are amongst the most popular algorithms in the reinforcement learning framework. Their advantage of being able to search for optimal policies using low-variance gradient estimates has made them useful in several real-life applications, such as robotics, power control, and finance. Although general surveys on reinforcement learning techniques already exist, no survey is specifically dedicated to actor-critic algorithms in particular. This paper, therefore, describes the state of the art of actor-critic algorithms, with a focus on methods that can work in an online setting and use function approximation in order to deal with continuous state and action spaces. After starting with a discussion on the concepts of reinforcement learning and the origins of actor-critic algorithms, this paper describes the workings of the natural gradient, which has made its way into many actor-critic algorithms over the past few years. A review of several standard and natural actor-critic algorithms is given, and the paper concludes with an overview of application areas and a discussion on open issues.

/pdf/a-survey-of-actor-critic-reinforcement-learning-standard-and-ya29rk9vmw.pdf

A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients

This paper investigates a novel model-free reinforcement learning architecture, the Natural Actor-Critic. The actor updates are based on stochastic policy gradients employing Amari's natural gradient approach, while the critic obtains both the natural policy gradient and additional parameters of a value function simultaneously by linear regression. We show that actor improvements with natural policy gradients are particularly appealing as these are independent of coordinate frame of the chosen policy representation, and can be estimated more efficiently than regular policy gradients. The critic makes use of a special basis function parameterization motivated by the policy-gradient compatible function approximation. We show that several well-known reinforcement learning methods such as the original Actor-Critic and Bradtke's Linear Quadratic Q-Learning are in fact Natural Actor-Critic algorithms. Empirical evaluations illustrate the effectiveness of our techniques in comparison to previous methods, and also demonstrate their applicability for learning control on an anthropomorphic robot arm.

Natural Actor-Critic

https://homes.cs.washington.edu/~todorov/courses/amath579/reading/NaturalActorCritic.pdf

Objectives This study compared adenosine-associated pleiotropic effects of the 2 P2Y12 receptor antagonists on vascular function, systemic inflammation, and circulating endothelial progenitor cells (EPCs). Background Both ticagrelor and prasugrel have potent antiplatelet effects. However, only ticagrelor inhibits cellular uptake of adenosine. Methods Using a randomized, crossover design with 10-week follow-up ticagrelor or prasugrel was administered to type 2 diabetic patients with non–ST-segment elevation acute coronary syndrome requiring stent implantation. A total of 62 patients underwent randomization in a 1:1 ratio to receive ticagrelor or prasugrel for 5 weeks followed by a direct cross over to the alternative treatment for 5 additional weeks. Brachial artery flow-mediated dilation, inflammatory markers, and number of circulating EPCs were compared. Results Improvement in brachial artery flow-mediated dilation was greater in the ticagrelor group (0.15 ± 0.19 mm vs. −0.03 ± 0.18 mm; p  Conclusions Compared with prasugrel, ticagrelor significantly decreased inflammatory cytokines such as interleukin 6 and tumor necrosis factor alpha and increased circulating EPCs, contributing to improved arterial endothelial function in diabetic non–ST-segment elevation acute coronary syndrome patients. Thus, data support that pleiotropic effects of ticagrelor beyond its potent antiplatelet effects could contribute to additional clinical benefits. (Comparison of Ticagrelor vs. Prasugrel on Inflammation, Arterial Stiffness, Endothelial Function, and Circulating Endothelial Progenitor Cells in Diabetic Patients With Non-ST Elevation Acute Coronary Syndrome [NSTE-ACS] Requiring Coronary Stenting; NCT02487732 )

Comparison of Ticagrelor Versus Prasugrel for Inflammation, Vascular Function, and Circulating Endothelial Progenitor Cells in Diabetic Patients With Non–ST-Segment Elevation Acute Coronary Syndrome Requiring Coronary Stenting: A Prospective, Randomized, Crossover Trial

Risk factors and outcomes of sepsis-induced myocardial dysfunction and stress-induced cardiomyopathy in sepsis or septic shock: A comparative retrospective study.

The support vector data description (SVDD) is one of the best-known one-class support vector learning methods, in which one tries the strategy of using balls defined on the feature space in order to distinguish a set of normal data from all other possible abnormal objects. The major concern of this letter is to extend the main idea of SVDD to pattern denoising. Combining the geodesic projection to the spherical decision boundary resulting from the SVDD, together with solving the preimage problem, we propose a new method for pattern denoising. We first solve SVDD for the training data and then for each noisy test pattern, obtain its denoised feature by moving its feature vector along the geodesic on the manifold to the nearest decision boundary of the SVDD ball. Finally we find the location of the denoised pattern by obtaining the pre-image of the denoised feature. The applicability of the proposed method is illustrated by a number of toy and real-world data sets.

/pdf/svdd-based-pattern-denoising-1mx1fixskv.pdf

SVDD-Based Pattern Denoising

The human body contains different endothelial cell types and differences in their angiogenic potential are poorly understood. We compared the functional angiogenic ability of human aortic endothelial cells (HAECs) and human umbilical vein endothelial cells (HUVECs) using a three-dimensional (3D) microfluidic cell culture system. HAECs and HUVECs exhibited similar cellular characteristics in a 2D culture system; however, in the 3D microfluidic angiogenesis system, HAECs exhibited stronger angiogenic potential than HUVECs. Interestingly, the expression level of fibroblast growth factor (FGF)2 and FGF5 under vascular endothelial growth factor (VEGF)-A stimulation was significantly higher in HAECs than in HUVECs. Moreover, small interfering RNA-mediated knockdown of FGF2 and FGF5 more significantly attenuated vascular sprouting induced from HAECs than HUVECs. Our results suggest that HAECs have greater angiogenic potential through FGF2 and FGF5 upregulation and could be a compatible endothelial cell type to achieve robust angiogenesis.

Jong-Ho Kim

Papers

Comparison of Ticagrelor Versus Prasugrel for Inflammation, Vascular Function, and Circulating Endothelial Progenitor Cells in Diabetic Patients With Non–ST-Segment Elevation Acute Coronary Syndrome Requiring Coronary Stenting: A Prospective, Randomized, Crossover Trial

Risk factors and outcomes of sepsis-induced myocardial dysfunction and stress-induced cardiomyopathy in sepsis or septic shock: A comparative retrospective study.

SVDD-Based Pattern Denoising

Intrinsic FGF2 and FGF5 promotes angiogenesis of human aortic endothelial cells in 3D microfluidic angiogenesis system.

Occurrence and genetic diversity of Blastocystis in Korean cattle.