As the size and service requirements of today’s networks gradually increase, large numbers of proprietary devices are deployed, which leads to network complexity, information security crises and makes network service and service provider management increasingly difficult. Network function virtualization (NFV) technology is one solution to this problem. NFV separates network functions from hardware and deploys them as software on a common server. NFV can be used to improve service flexibility and isolate the services provided for each user, thus guaranteeing the security of user data. Therefore, the use of NFV technology includes many problems worth studying. For example, when there is a free choice of network path, one problem is how to choose a service function chain (SFC) that both meets the requirements and offers the service provider maximum profit. Most existing solutions are heuristic algorithms with high time efficiency, or integer linear programming (ILP) algorithms with high accuracy. It’s necessary to design an algorithm that symmetrically considers both time efficiency and accuracy. In this paper, we propose the Q-learning Framework Hybrid Module algorithm (QLFHM), which includes reinforcement learning to solve this SFC deployment problem in dynamic networks. The reinforcement learning module in QLFHM is responsible for the output of alternative paths, while the load balancing module in QLFHM is responsible for picking the optimal solution from them. The results of a comparison simulation experiment on a dynamic network topology show that the proposed algorithm can output the approximate optimal solution in a relatively short time while also considering the network load balance. Thus, it achieves the goal of maximizing the benefit to the service provider.

/pdf/a-q-learning-based-approach-for-deploying-dynamic-service-49cefg9q8k.pdf

A Q-learning-based approach for deploying dynamic service function chains

Background and Objectives: IP multimedia subsystems (IMS) have been introduced as the Next Generation Network (NGN) platform while considering Session Initiation Protocol (SIP) as the signaling protocol. SIP lacks a proper overload mechanism. Hence, this challenge causes decline in the multimedia QoS. The main propose of overload control mechanism is to keep the network throughput at the same network capacity with overload. Methods: NGN distributed with IMS is a complex innovative network consisting of interacting subsystems. Hence, multi-agent systems (MAS) receiving further attention for solving complex problems can solve the problem of overload in these networks. To this end, each IMS server is considered as an intelligent agent that can learn and negotiate with other agents while maintaining autonomy, thus eliminating the overload by communication and knowledge transfer between the agents. In the present research, using MAS and their properties, the intelligent hop by hop method is provided based on Q-learning and negotiation capability for the first time. Results: In the proposed method, parameters of overload controller are obtained by reinforcement learning. In order to check the validity of controller performance, a comparison is made with the similar method in which the optimal parameters are achieved based on trial and error. The result of the comparison confirms the validity of the proposed method. In order to evaluate the efficiency of the learner method, it is compared with similar and standard methods, for which the results are compared to show performance. The results show, the proposed method has approximately improved the throughput by 13%, the delay by 49% and the number of rejected sessions by 17% compare with methods, passing control messages through the network such as CPU occupancy methods. While compare with external controller methods like holonic, throughput is improved by 1% and the number of rejected requests is decreased by 10%, but delay is increased by 6% due to the convergence time of the learning and negotiation process. Conclusion: To overcome overload, complex IMS servers are considered as learner and negotiator agents. This is a new method to achieve the required parameters without relying on expert knowledge or person as well as, heterogeneous IMS entities can be inserted into the problem to complete study in future.

Reinforcement Learning-based Load Controller in IP Multimedia Subsystems

Session Initiation Protocol (SIP) is considered as a signaling protocol for IP multimedia subsystem (IMS) IMS is introduced by 3rd generation partnership project (3GPP) as signaling foundation in next generation networks (NGN) Despite having the features such as: text based, IP based, independent of the data transmission, support for mobility and end-to-end, the SIP protocol has not suitable mechanism to deal with overload Therefore, many mechanisms are proposed to control overload in SIP networks One of their most famous is occupancy CPU (OCC) that is used in many researches In traditional OCC, the value of parameters is indicated and they are used in subsequent documents In this paper, optimal parameters value is obtained by Q-learning algorithm Because modeling a large SIP network is impossible by mathematical relationships and it is a heuristic problem, Q-learning is the best method to compute the parameters The simulation results demonstrate that the Q-learning output is comparable with traditional OCC

Mehdi Khazaei

Papers

Occupancy Overload Control by Q-learning