Dimensions of Reinforcement Learning

Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design. However, existing MCTS based NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy), leading to sample-inefficient explorations of architectures. To improve the sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with similar performance metrics. During the search phase, as different action sequences lead to regions with different performance, the search efficiency can be significantly improved by biasing towards the good regions. On three NAS tasks, empirical results demonstrate that LaNAS is at least an order more sample efficient than baseline methods including evolutionary algorithms, Bayesian optimizations, and random search. When applied in practice, both one-shot and regular LaNAS consistently outperform existing results. Particularly, LaNAS achieves 99.0% accuracy on CIFAR-10 and 80.8% top1 accuracy at 600 MFLOPS on ImageNet in only 800 samples, significantly outperforming AmoebaNet with 33x fewer samples. Our code is publicly available at this https URL.

Sample-Efficient Neural Architecture Search by Learning Action Space.

Many important applications, including robotics, data-center management, and process control, require planning action sequences in domains with continuous state and action spaces and discontinuous objective functions. Monte Carlo tree search (MCTS) is an effective strategy for planning in discrete action spaces. We provide a novel MCTS algorithm (voot) for deterministic environments with continuous action spaces, which, in turn, is based on a novel black-box function-optimization algorithm (voo) to efficiently sample actions. The voo algorithm uses Voronoi partitioning to guide sampling, and is particularly efficient in high-dimensional spaces. The voot algorithm has an instance of voo at each node in the tree. We provide regret bounds for both algorithms and demonstrate their empirical effectiveness in several high-dimensional problems including two difficult robotics planning problems.

/pdf/monte-carlo-tree-search-in-continuous-spaces-using-voronoi-pv845i6w33.pdf

Monte Carlo Tree Search in Continuous Spaces Using Voronoi Optimistic Optimization with Regret Bounds

Control of nonlinear systems is challenging in realtime. Decision making, performed many times per second, must ensure system safety. Designing input to perform a task often involves solving a nonlinear system of differential equations, which is a computationally intensive, if not intractable problem. This article proposes sampling-based task learning for control-affine nonlinear systems through the combined learning of both state and action-value functions in a model-free approximate value iteration setting with continuous inputs. A quadratic negative definite state-value function implies the existence of a unique maximum of the action-value function at any state. This allows the replacement of the standard greedy policy with a computationally efficient policy approximation that guarantees progression to a goal state without knowledge of the system dynamics. The policy approximation is consistent, i.e., it does not depend on the action samples used to calculate it. This method is appropriate for mechanical systems with high-dimensional input spaces and unknown dynamics performing Constraint-Balancing Tasks. We verify it both in simulation and experimentally for an Unmanned Aerial Vehicles (UAVs) carrying a suspended load, and in simulation, for the rendezvous of heterogeneous robots.

http://ieeexplore.ieee.org/iel7/6570654/7004676/07004690.pdf

Continuous action reinforcement learning for control-affine systems with unknown dynamics

This paper considers global optimization with a black-box unknown objective function that can be non-convex and non-differentiable. Such a difficult optimization problem arises in many real-world applications, such as parameter tuning in machine learning, engineering design problem, and planning with a complex physics simulator. This paper proposes a new global optimization algorithm, called Locally Oriented Global Optimization (LOGO), to aim for both fast convergence in practice and finite-time error bound in theory. The advantage and usage of the new algorithm are illustrated via theoretical analysis and an experiment conducted with 11 benchmark test functions. Further, we modify the LOGO algorithm to specifically solve a planning problem via policy search with continuous state/action space and long time horizon while maintaining its finite-time error bound. We apply the proposed planning method to accident management of a nuclear power plant. The result of the application study demonstrates the practical utility of our method.

/pdf/global-continuous-optimization-with-error-bound-and-fast-4gpjnjzblq.pdf

Global continuous optimization with error bound and fast convergence

We consider the class of online planning algorithms for optimal control, which compared to dynamic programming are relatively unaffected by large state dimensionality. We introduce a novel planning algorithm called SOOP that works for deterministic systems with continuous states and actions. SOOP is the first method to explore the true solution space, consisting of infinite sequences of continuous actions, without requiring knowledge about the smoothness of the system. SOOP can be used parameter-free at the cost of more model calls, but we also propose a more practical variant tuned by a parameter α, which balances finer discretization with longer planning horizons. Experiments on three problems show SOOP reliably ranks among the best algorithms, fully dominating competing methods when the problem requires both long horizons and fine discretization.

/pdf/optimistic-planning-for-continuous-action-deterministic-59pmtthwfz.pdf

Optimistic planning for continuous-action deterministic systems

Online learning for optimistic planning

Random networks, constrained to reproduce speciﬁc features of networks, are often used to represent and analyze network data as well as their mathematical descriptions. Chief among them, the conﬁguration model constrains random networks by their degree distribution and is foundational to many areas of network science. However, these representations are often selected based on intuition or mathematical and computational simplicity rather than on statistical evidence. To evaluate the quality of a network representation we need to consider both the amount of information required by a random network model as well as the probability of recovering the original data when using the model as a generative process. To this end, we calculate the approximate size of network ensembles generated by the popular conﬁguration model and its generalizations that include degree-correlations and centrality layers based on the onion decomposition. We then apply minimum description length as a model selection criterion and also introduce the Network Onion Divergence: model selection and network comparison over a nested family of conﬁguration models with diﬀering level of structural details. Using over 100 empirical sets of network data, we ﬁnd that a simple Layered Conﬁguration Model oﬀers the most compact representation of the majority of real networks. We hope that our results will continue to motivate the development of intricate random network models that help capture network structure beyond the simple degree distribution. food webs), infrastructure (e.g. roads, power grids, ﬂights), social (e.g. co-authorship, social media friendships), or technological networks (e.g. world-wide web subsets). We do not expect any signiﬁcant signal as empirical networks can vary greatly in structure, but this simple experiment again illustrates how the LCM divergence makes a better use of the [0 , 1] interval and provides more contrast than other divergences.

Network Onion Divergence: Network representation and comparison using nested configuration models with fixed connectivity, correlation and centrality patterns

Alexander Daniels

Papers

Optimistic planning for continuous-action deterministic systems

Online learning for optimistic planning

Network Onion Divergence: Network representation and comparison using nested configuration models with fixed connectivity, correlation and centrality patterns