This paper reviews the current state of the art on reinforcement learning (RL)-based feedback control solutions to optimal regulation and tracking of single and multiagent systems. Existing RL solutions to both optimal $\mathcal {H}_{2}$ and $\mathcal {H}_\infty $ control problems, as well as graphical games, will be reviewed. RL methods learn the solution to optimal control and game problems online and using measured data along the system trajectories. We discuss Q-learning and the integral RL algorithm as core algorithms for discrete-time (DT) and continuous-time (CT) systems, respectively. Moreover, we discuss a new direction of off-policy RL for both CT and DT systems. Finally, we review several applications.

Optimal and Autonomous Control Using Reinforcement Learning: A Survey

ResearchGate is increasingly used by scholars to upload the full-text of their articles and make them freely available for everyone. This study aims to investigate the extent to which ResearchGate members as authors of journal articles comply with publishers' copyright policies when they self-archive full-text of their articles on ResearchGate. A random sample of 500 English journal articles available as full-text on ResearchGate were investigated. 108 articles (21.6%) were open access (OA) published in OA journals or hybrid journals. Of the remaining 392 articles, 61 (15.6%) were preprint, 24 (6.1%) were post-print and 307 (78.3%) were published (publisher) PDF. The key finding was that 201 (51.3%) out of 392 non-OA articles infringed the copyright and were non-compliant with publishers' policy. While 88.3% of journals allowed some form of self-archiving (SHERPA/RoMEO green, blue or yellow journals), the majority of non-compliant cases (97.5%) occurred when authors self-archived publishers' PDF files (final published version). This indicates that authors infringe copyright most of the time not because they are not allowed to self-archive, but because they use the wrong version, which might imply their lack of understanding of copyright policies and/or complexity and diversity of policies.

Copyright compliance and infringement in ResearchGate full-text journal articles

Large software companies have to plan their project portfolio to maximize potential portfolio return and strategic alignment, while balancing various preferences, and considering limited resources. Project portfolio managers need methods and tools to find a good solution for complex project portfolios and multiobjective target criteria efficiently. However, software project portfolios are challenging to describe for optimization in a practical way that allows efficient optimization. In this paper we propose an approach to describe software project portfolios with a set of multiobjective criteria for portfolio managers using the COCOMO II model and introduce a multiobjective evolutionary approach, mPOEMS, to find the Pareto-optimal front efficiently. We evaluate the new approach with portfolios choosing from a set of 50 projects that follow the validated COCOMO II model criteria and compare the performance of the mPOEMS approach with state-of-the-art multiobjective optimization evolutionary approaches. Major results are as follows: the portfolio management approach was found usable and useful; the mPOEMS approach outperformed the other approaches.

/pdf/software-project-portfolio-optimization-with-advanced-1q9o5a9up2.pdf

Software project portfolio optimization with advanced multiobjective evolutionary algorithms

Experience replay is a technique that allows off-policy reinforcement-learning methods to reuse past experiences. The stability and speed of convergence of reinforcement learning, as well as the eventual performance of the learned policy, are strongly dependent on the experiences being replayed. Which experiences are replayed depends on two important choices. The first is which and how many experiences to retain in the experience replay buffer. The second choice is how to sample the experiences that are to be replayed from that buffer. We propose new methods for the combined problem of experience retention and experience sampling. We refer to the combination as experience selection. We focus our investigation specifically on the control of physical systems, such as robots, where exploration is costly. To determine which experiences to keep and which to replay, we investigate different proxies for their immediate and long-term utility. These proxies include age, temporal difference error and the strength of the applied exploration noise. Since no currently available method works in all situations, we propose guidelines for using prior knowledge about the characteristics of the control problem at hand to choose the appropriate experience replay strategy.

/pdf/experience-selection-in-deep-reinforcement-learning-for-3bxri51gme.pdf

Experience selection in deep reinforcement learning for control

Effective knowledge exchange among software developers is crucial for the competitive performance of their organizations. Today, the constant pressure on businesses to continually innovate and the increasing capability of information technologies to facilitate broader and more distributed communication are driving organizations to leverage social media tools to improve performance. These tools, which have changed the way we share knowledge, enable people to connect, communicate, and collaborate. Research on knowledge sharing via social media is still in its early phases, with a comprehensive overview of the literature yet to be completed. Thus, using a systematic literature review approach, this study aims to map the current literature on the topic in relation to software development. Furthermore, this study highlights the findings of former research and identifies gaps in the literature. The study offers several insights for researchers and practitioners and proposes a future research agenda to strengthen knowledge in the field.

/pdf/knowledge-sharing-via-social-media-in-software-development-a-13jv99jj89.pdf

Knowledge sharing via social media in software development: a systematic literature review

This paper addresses the problem of deriving a policy from the value function in the context of reinforcement learning in continuous state and input spaces. We propose a novel method based on genetic programming to construct a symbolic function, which serves as a proxy to the value function and from which a continuous policy is derived. The symbolic proxy function is constructed such that it maximizes the number of correct choices of the control input for a set of selected states. Maximization methods can then be used to derive a control policy that performs better than the policy derived from the original approximate value function. The method was experimentally evaluated on two control problems with continuous spaces, pendulum swing-up and magnetic manipulation, and compared to a standard policy derivation method using the value function approximation. The results show that the proposed method and its variants outperform the standard method.

Symbolic method for deriving policy in reinforcement learning

It is well known that reinforcement learning (RL) can benefit from the use of a dynamic prediction model which is learned on data samples collected online from the process to be controlled. Most RL algorithms are formulated in the state-space domain and use state-space models. However, learning state-space models is difficult, mainly because in the vast majority of problems the full state cannot be measured on the system or reconstructed from the measurements. To circumvent this limitation, we propose to use input-output models of the NARX (nonlinear autoregressive with exogenous input) type. Symbolic regression is employed to construct parsimonious models and the corresponding value functions. Thanks to this approach, we can learn accurate models and compute optimal policies even from small amounts of training data. We demonstrate the approach on two simulated examples, a hopping robot and a 1-DOF robot arm, and on a real inverted pendulum system. Results show that our proposed method can reliably determine a good control policy based on a symbolic input-output process model and value function.

Reinforcement Learning with Symbolic Input-Output Models

Genetic programming (GP) is a technique widely used in a range of symbolic regression problems, in particular when there is no prior knowledge about the symbolic function sought. In this paper, we present a GP extension introducing a new concept of local transformed variables, based on a locally applied affine transformation of the original variables. This approach facilitates finding accurate parsimonious models. We have evaluated the proposed extension in the context of the Single Node Genetic Programming (SNGP) algorithm on synthetic as well as real-problem datasets. The results confirm our hypothesis that the transformed variables significantly improve the performance of the standard SNGP algorithm.

Enhanced Symbolic Regression Through Local Variable Transformations

Many real-world problems can be naturally described by mathematical formulas. The task of finding formulas from a set of observed inputs and outputs is called symbolic regression. Recently, neural networks have been applied to symbolic regression, among which the transformer-based ones seem to be the most promising. After training the transformer on a large number of formulas (in the order of days), the actual inference, i.e., finding a formula for new, unseen data, is very fast (in the order of seconds). This is considerably faster than state-of-the-art evolutionary methods. The main drawback of transformers is that they generate formulas without numerical constants, which have to be optimized separately, so yielding suboptimal results. We propose a transformer-based approach called SymFormer, which predicts the formula by outputting the individual symbols and the corresponding constants simultaneously. This leads to better performance in terms of fitting the available data. In addition, the constants provided by SymFormer serve as a good starting point for subsequent tuning via gradient descent to further improve the performance. We show on a set of benchmarks that SymFormer outperforms two state-of-the-art methods while having faster inference.

Jiri Kubalik

Papers

Symbolic method for deriving policy in reinforcement learning

Reinforcement Learning with Symbolic Input-Output Models

Enhanced Symbolic Regression Through Local Variable Transformations

SymFormer: End-to-end symbolic regression using transformer-based architecture

Policy Derivation Methods for Critic-Only Reinforcement Learning in Continuous Action Spaces