Deep Q-Learning Aided Networking, Caching, and Computing Resources Allocation in Software-Defined Satellite-Terrestrial Networks
TL;DR: This paper proposes a software-defined STN to manage and orchestrate networking, caching, and computing resources jointly, and forms the joint resources allocation problem as a joint optimization problem, and uses a deep Q-learning approach to solve it.
Abstract: With the development of satellite networks, there is an emerging trend to integrate satellite networks with terrestrial networks, called satellite-terrestrial networks (STNs). The improvements of STNs need innovative information and communication technologies, such as networking, caching, and computing. In this paper, we propose a software-defined STN to manage and orchestrate networking, caching, and computing resources jointly. We formulate the joint resources allocation problem as a joint optimization problem, and use a deep Q-learning approach to solve it. Simulation results show the effectiveness of our proposed scheme.
Citations
More filters
TL;DR: This survey guides the reader through a comprehensive discussion of the main characteristics of SDN and NFV technologies, and provides a thorough analysis of the different classifications, use cases, and challenges related to UAV-assisted systems.
Abstract: Unmanned Aerial Vehicles (UAVs) have become increasingly important in assisting 5G and beyond 5G (B5G) mobile networks. Indeed, UAVs have all the potentials to both satisfy the ever-increasing mobile data demands of such mobile networks and provide ubiquitous connectivity to different kinds of wireless devices. However, the UAV assistance paradigm faces a set of crucial issues and challenges. For example, the network management of current UAV-assisted systems is time consuming, complicated, and carried out manually, thus causing a multitude of interoperability issues. To efficiently address all these issues, Software-Defined Network (SDN) and Network Function Virtualization (NFV) are two promising technologies to efficiently manage and improve the UAV assistance for the next generation of mobile networks. In the literature, no clear guidelines are describing the different use cases of SDN and NFV in the context of UAV assistance to terrestrial networks, including mobile networks. Motivated by this fact, in this survey, we guide the reader through a comprehensive discussion of the main characteristics of SDN and NFV technologies. Moreover, we provide a thorough analysis of the different classifications, use cases, and challenges related to UAV-assisted systems. We then discuss SDN/NFV-enabled UAV-assisted systems, along with several case studies and issues, such as the involvement of UAVs in cellular communications, monitoring, and routing, to name a few. We furthermore present a set of open research challenges, high-level insights, and future research directions related to UAV-assisted systems.
137 citations
TL;DR: This paper describes ARAN architecture and its fundamental features for the development of 6G networks, and introduces technologies that enable the success of ARAN implementations in terms of energy replenishment, operational management, and data delivery.
Abstract: Current access infrastructures are characterized by heterogeneity, low latency, high throughput, and high computational capability, enabling massive concurrent connections and various services. Unfortunately, this design does not pay significant attention to mobile services in underserved areas. In this context, the use of aerial radio access networks (ARANs) is a promising strategy to complement existing terrestrial communication systems. Involving airborne components such as unmanned aerial vehicles, drones, and satellites, ARANs can quickly establish a flexible access infrastructure on demand. ARANs are expected to support the development of seamless mobile communication systems toward a comprehensive sixth-generation (6G) global access infrastructure. This paper provides an overview of recent studies regarding ARANs in the literature. First, we investigate related work to identify areas for further exploration in terms of recent knowledge advancements and analyses. Second, we define the scope and methodology of this study. Then, we describe ARAN architecture and its fundamental features for the development of 6G networks. In particular, we analyze the system model from several perspectives, including transmission propagation, energy consumption, communication latency, and network mobility. Furthermore, we introduce technologies that enable the success of ARAN implementations in terms of energy replenishment, operational management, and data delivery. Subsequently, we discuss application scenarios envisioned for these technologies. Finally, we highlight ongoing research efforts and trends toward 6G ARANs.
136 citations
TL;DR: This article proposes a distributed algorithm by leveraging the alternating direction method of multipliers (ADMMs) to approximate the optimal solution with low computational complexity and shows that the proposed algorithm can effectively reduce the total energy consumption of ground users.
Abstract: Low earth orbit (LEO) satellite networks can break through geographical restrictions and achieve global wireless coverage, which is an indispensable choice for future mobile communication systems In this article, we present a hybrid cloud and edge computing LEO satellite (CECLS) network with a three-tier computation architecture, which can provide ground users with heterogeneous computation resources and enable ground users to obtain computation services around the world With the CECLS architecture, we investigate the computation offloading decisions to minimize the sum energy consumption of ground users, while satisfying the constraints in terms of the coverage time and the computation capability of each LEO satellite The considered problem leads to a discrete and nonconvex since the objective function and constraints contain binary variables, which makes it difficult to solve To address this challenging problem, we convert the original nonconvex problem into a linear programming problem by using the binary variables relaxation method Then, we propose a distributed algorithm by leveraging the alternating direction method of multipliers (ADMMs) to approximate the optimal solution with low computational complexity Simulation results show that the proposed algorithm can effectively reduce the total energy consumption of ground users
100 citations
Cites background or methods from "Deep Q-Learning Aided Networking, C..."
...Compared with the researches in [17]–[21], where only a two-tier computing network including terrestrial networks (i....
[...]
...[21] presented a software-defined satelliteterrestrial network to dynamically manage the caching and computing resources of satellite-terrestrial networks, where a deep Q-learning algorithm is developed to solve the joint resource allocation optimization problem....
[...]
TL;DR: A model of the three-layer heterogeneous satellite network is constructed and a low-complexity method for calculating the capacity between satellites is proposed and a long-term optimal capacity allocation algorithm is proposed to optimize the long- term utility of the system.
Abstract: The development of satellite networks is drawing much more attention in recent years due to the wide coverage ability. Composed of geosynchronous orbit (GEO), medium earth orbit (MEO), and low earth orbit (LEO) satellites, the satellite network is a three-layer heterogeneous network of high complexity, for which comprehensive theoretical analysis is still missing. In this paper, we investigate the problem of capacity management in the three-layer heterogeneous satellite network. We first construct the model of the network and propose a low-complexity method for calculating the capacity between satellites. Based on the time structure of the time expanded graph, the searching space is greatly reduced compared to traditional augmenting path searching strategies, which can significantly reduce the computing complexity. Then, based on Q-learning, we proposed a long-term optimal capacity allocation algorithm to optimize the long-term utility of the system. In order to reduce the storage and computing complexity, a learning framework with low-complexity is constructed while taking the properties of satellite systems into account. Finally, we analyze the capacity performance of the three-layer heterogeneous satellite network and also evaluate the proposed algorithms with numerical results.
99 citations
Cites methods from "Deep Q-Learning Aided Networking, C..."
...In this paper, we use Q-Learning to solve the dynamic capacity allocation problem, which is a type of reinforcement learning proposed by Watkins [27] and Watkins and Dayan [28], and has been applied in many different communication systems for resource allocation [29]–[33]....
[...]
TL;DR: The detailed functional components of the proposed STECN are discussed, and the promising technical challenges, including meeting QoE requirements, cooperative computation offloading, multi-node task scheduling, mobility management and fault/failure recovery are presented.
Abstract: STN has been considered a novel network architecture to accommodate a variety of services and applications in future networks. Being a promising paradigm, MEC has been regarded as a key technology-enabler to offer further service innovation and business agility in STN. However, most of the existing research in MEC enabled STN regards a satellite network as a relay network, and the feasibility of tasks processing directly on the satellites is largely ignored. Moreover, the problem of multi-layer edge computing architecture design and heterogeneous edge computing resource co-scheduling, have not been fully considered. Therefore, different from previous works, in this article, we propose a novel architecture named STECN, in which computing resources exist in multi-layer heterogeneous edge computing clusters. The detailed functional components of the proposed STECN are discussed, and we present the promising technical challenges, including meeting QoE requirements, cooperative computation offloading, multi-node task scheduling, mobility management and fault/failure recovery. Finally, some potential research issues for future research are highlighted.
99 citations
Cites methods from "Deep Q-Learning Aided Networking, C..."
...In [14], the authors develop a software-defined STN to manage and coordinate the network, caching and computing resources, and design a deep learning method to solve the joint resource allocation problem....
[...]
References
More filters
TL;DR: This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
Abstract: The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
23,074 citations
"Deep Q-Learning Aided Networking, C..." refers background in this paper
...There are two innovations to make DQL more efficient and more robust [44]: 1) Experience replay....
[...]
02 Nov 2016
TL;DR: TensorFlow as mentioned in this paper is a machine learning system that operates at large scale and in heterogeneous environments, using dataflow graphs to represent computation, shared state, and the operations that mutate that state.
Abstract: TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. Tensor-Flow uses dataflow graphs to represent computation, shared state, and the operations that mutate that state. It maps the nodes of a dataflow graph across many machines in a cluster, and within a machine across multiple computational devices, including multicore CPUs, general-purpose GPUs, and custom-designed ASICs known as Tensor Processing Units (TPUs). This architecture gives flexibility to the application developer: whereas in previous "parameter server" designs the management of shared state is built into the system, TensorFlow enables developers to experiment with novel optimizations and training algorithms. TensorFlow supports a variety of applications, with a focus on training and inference on deep neural networks. Several Google services use TensorFlow in production, we have released it as an open-source project, and it has become widely used for machine learning research. In this paper, we describe the TensorFlow dataflow model and demonstrate the compelling performance that TensorFlow achieves for several real-world applications.
10,913 citations
31 Mar 2008
TL;DR: This whitepaper proposes OpenFlow: a way for researchers to run experimental protocols in the networks they use every day, based on an Ethernet switch, with an internal flow-table, and a standardized interface to add and remove flow entries.
Abstract: This whitepaper proposes OpenFlow: a way for researchers to run experimental protocols in the networks they use every day. OpenFlow is based on an Ethernet switch, with an internal flow-table, and a standardized interface to add and remove flow entries. Our goal is to encourage networking vendors to add OpenFlow to their switch products for deployment in college campus backbones and wiring closets. We believe that OpenFlow is a pragmatic compromise: on one hand, it allows researchers to run experiments on heterogeneous switches in a uniform way at line-rate and with high port-density; while on the other hand, vendors do not need to expose the internal workings of their switches. In addition to allowing researchers to evaluate their ideas in real-world traffic settings, OpenFlow could serve as a useful campus component in proposed large-scale testbeds like GENI. Two buildings at Stanford University will soon run OpenFlow networks, using commercial Ethernet switches and routers. We will work to encourage deployment at other schools; and We encourage you to consider deploying OpenFlow in your university network too
9,138 citations
"Deep Q-Learning Aided Networking, C..." refers background in this paper
...networking resources from LEOs, caching resources from content caches, and computing resources from MEC servers) by OpenFlow messages [39], so as to meet users’ requirements....
[...]
TL;DR: This paper is the first to present the state-of-the-art of the SAGIN since existing survey papers focused on either only one single network segment in space or air, or the integration of space-ground, neglecting the Integration of all the three network segments.
Abstract: Space-air-ground integrated network (SAGIN), as an integration of satellite systems, aerial networks, and terrestrial communications, has been becoming an emerging architecture and attracted intensive research interest during the past years. Besides bringing significant benefits for various practical services and applications, SAGIN is also facing many unprecedented challenges due to its specific characteristics, such as heterogeneity, self-organization, and time-variability. Compared to traditional ground or satellite networks, SAGIN is affected by the limited and unbalanced network resources in all three network segments, so that it is difficult to obtain the best performances for traffic delivery. Therefore, the system integration, protocol optimization, resource management, and allocation in SAGIN is of great significance. To the best of our knowledge, we are the first to present the state-of-the-art of the SAGIN since existing survey papers focused on either only one single network segment in space or air, or the integration of space-ground, neglecting the integration of all the three network segments. In light of this, we present in this paper a comprehensive review of recent research works concerning SAGIN from network design and resource allocation to performance analysis and optimization. After discussing several existing network architectures, we also point out some technology challenges and future directions.
661 citations
"Deep Q-Learning Aided Networking, C..." refers background in this paper
...Thus, there is an emerging trend to integrate satellite networks with terrestrial networks, called satellite-terrestrial networks (STNs) [4], [5]....
[...]
TL;DR: A software defined spaceair- ground integrated network architecture for supporting diverse vehicular services in a seamless, efficient, and cost-effective manner is proposed.
Abstract: This article proposes a software defined spaceair- ground integrated network architecture for supporting diverse vehicular services in a seamless, efficient, and cost-effective manner. First, the motivations and challenges for integration of space-air-ground networks are reviewed. Second, a software defined network architecture with a layered structure is presented. To protect the legacy services in the satellite, aerial, and terrestrial segments, resources in each segment are sliced through network slicing to achieve service isolation. Then available resources are put into a common and dynamic space-air-ground resource pool, which is managed by hierarchical controllers to accommodate vehicular services. Finally, a case study is carried out, followed by discussion on some open research topics.
339 citations
"Deep Q-Learning Aided Networking, C..." refers background in this paper
...Moreover, network virtualization enables to virtualize a single physical network into several virtual networks to share networking resources [11], which improves the effectiveness of networking resources in STNs....
[...]