Showing papers on "Job shop published in 2022"

Dynamic job shop scheduling based on deep reinforcement learning for multi-agent manufacturing systems

Abstract: Personalized orders bring challenges to the production paradigm, and there is an urgent need for the dynamic responsiveness and self-adjustment ability of the workshop. Traditional dispatching rules and heuristic algorithms solve the production planning and control problems by making schedules. However, the previous methods cannot work well in a changeable workshop environment when encountering a large number of stochastic disturbances of orders and resources. Recently, the potential of artificial intelligence (AI) algorithms in solving the dynamic scheduling problem has attracted researchers' attention. Therefore, this paper presents a multi-agent manufacturing system based on deep reinforcement learning (DRL), which integrates the self-organization mechanism and self-learning strategy. Firstly, the manufacturing equipment in the workshop is constructed as an equipment agent with the support of edge computing node, and an improved contract network protocol (CNP) is applied to guide the cooperation and competition among multiple agents, so as to complete personalized orders efficiently. Secondly, a multi-layer perceptron is employed to establish the decision-making module called AI scheduler inside the equipment agent. According to the perceived workshop state information, AI scheduler intelligently generates an optimal production strategy to perform task allocation. Then, based on the collected sample trajectories of scheduling process, AI scheduler is periodically trained and updated through the proximal policy optimization (PPO) algorithm to improve its decision-making performance. Finally, in the multi-agent manufacturing system testbed, dynamic events such as stochastic job insertions and unpredictable machine failures are considered in the verification experiments. The experimental results show that the proposed method is capable of obtaining the scheduling solutions that meet various performance metrics, as well as dealing with resource or task disturbances efficiently and autonomously.

Deep reinforcement learning for dynamic scheduling of a flexible job shop

Abstract: The ability to handle unpredictable dynamic events is becoming more important in pursuing agile and flexible production scheduling. At the same time, the cyber-physical convergence in production system creates massive amounts of industrial data that needs to be mined and analysed in real-time. To facilitate such real-time control, this research proposes a hierarchical and distributed architecture to solve the dynamic flexible job shop scheduling problem. Double Deep Q-Network algorithm is used to train the scheduling agents, to capture the relationship between production information and scheduling objectives, and make real-time scheduling decisions for a flexible job shop with constant job arrivals. Specialised state and action representations are proposed to handle the variable specification of the problem in dynamic scheduling. Additionally, a surrogate reward-shaping technique to improve learning efficiency and scheduling effectiveness is developed. A simulation study is carried out to validate the performance of the proposed approach under different scenarios. Numerical results show that not only does the proposed approach deliver superior performance as compared to existing scheduling strategies, its advantages persist even if the manufacturing system configuration changes.

Real-time data-driven dynamic scheduling for flexible job shop with insufficient transportation resources using hybrid deep Q network

Abstract: With the extensive application of automated guided vehicles in manufacturing system, production scheduling considering limited transportation resources becomes a difficult problem. At the same time, the real manufacturing system is prone to various disturbance events, which increase the complexity and uncertainty of shop floor. To this end, this paper addresses the dynamic flexible job shop scheduling problem with insufficient transportation resources (DFJSP-ITR) to minimize the makespan and total energy consumption. As a sequential decision-making problem, DFJSP-ITR can be modeled as a Markov decision process where the agent should determine the scheduling object and allocation of resources at each decision point. So this paper adopts deep reinforcement learning to solve DFJSP-ITR. In this paper, the multiobjective optimization model of DFJSP-ITR is established. Then, in order to make agent learn to choose the appropriate rule based on the production state at each decision point, a hybrid deep Q network (HDQN) is developed for this problem, which combines deep Q network with three extensions. Moreover, the shop floor state model is established at first, and then the decision point, generic state features, genetic-programming-based action space and reward function are designed. Based on these contents, the training method using HDQN and the strategy for facing new job insertions and machine breakdowns are proposed. Finally, comprehensive experiments are conducted, and the results show that HDQN has superiority and generality compared with current optimization-based approaches, and can effectively deal with disturbance events and unseen situations through learning.

Real-time data-driven dynamic scheduling for flexible job shop with insufficient transportation resources using hybrid deep Q network

Abstract: With the extensive application of automated guided vehicles in manufacturing system, production scheduling considering limited transportation resources becomes a difficult problem. At the same time, the real manufacturing system is prone to various disturbance events, which increase the complexity and uncertainty of shop floor. To this end, this paper addresses the dynamic flexible job shop scheduling problem with insufficient transportation resources (DFJSP-ITR) to minimize the makespan and total energy consumption. As a sequential decision-making problem, DFJSP-ITR can be modeled as a Markov decision process where the agent should determine the scheduling object and allocation of resources at each decision point. So this paper adopts deep reinforcement learning to solve DFJSP-ITR. In this paper, the multiobjective optimization model of DFJSP-ITR is established. Then, in order to make agent learn to choose the appropriate rule based on the production state at each decision point, a hybrid deep Q network (HDQN) is developed for this problem, which combines deep Q network with three extensions. Moreover, the shop floor state model is established at first, and then the decision point, generic state features, genetic-programming-based action space and reward function are designed. Based on these contents, the training method using HDQN and the strategy for facing new job insertions and machine breakdowns are proposed. Finally, comprehensive experiments are conducted, and the results show that HDQN has superiority and generality compared with current optimization-based approaches, and can effectively deal with disturbance events and unseen situations through learning.

Deep Reinforcement Learning for Dynamic Flexible Job Shop Scheduling with Random Job Arrival

Abstract: The production process of a smart factory is complex and dynamic. As the core of manufacturing management, the research into the flexible job shop scheduling problem (FJSP) focuses on optimizing scheduling decisions in real time, according to the changes in the production environment. In this paper, deep reinforcement learning (DRL) is proposed to solve the dynamic FJSP (DFJSP) with random job arrival, with the goal of minimizing penalties for earliness and tardiness. A double deep Q-networks (DDQN) architecture is proposed and state features, actions and rewards are designed. A soft ε-greedy behavior policy is designed according to the scale of the problem. The experimental results show that the proposed DRL is better than other reinforcement learning (RL) algorithms, heuristics and metaheuristics in terms of solution quality and generalization. In addition, the soft ε-greedy strategy reasonably balances exploration and exploitation, thereby improving the learning efficiency of the scheduling agent. The DRL method is adaptive to the dynamic changes of the production environment in a flexible job shop, which contributes to the establishment of a flexible scheduling system with self-learning, real-time optimization and intelligent decision-making.

Real-time scheduling simulation optimisation of job shop in a production-logistics collaborative environment

Abstract: ABSTRACT In a complex and dynamic job shop containing logistics factor, schedule needs to be generated rapidly, so the real-time scheduling method is more suitable for such scenario. Such method takes advantage of local information within a short time due to the rapid changes of information under uncertain environment. Therefore, how to make use of the future information by prediction while ensuring the robustness of schedule is a valuable problem. To solve it, firstly, a new real-time scheduling model and algorithm is proposed. There is a new kind of release moment of task information which can give AGVs the longest time to prepare for the task than existing research. Secondly, a real-time information update mechanism is designed to increase schedule’s robustness. Finally, a large-scale and dynamic job shop simulation experimental platform is developed. Dynamic factors include the random insertion of orders and failures of equipment. Results show that the method proposed outperforms existing research in terms of customer satisfaction, equipment utilisation and energy consumption. The robustness of schedule can also be acceptable. This paper also finds a rule that in job shop with the large proportion of logistics transportation time, the above method can achieve more competitive results.

Real-Time Scheduling for Dynamic Partial-No-Wait Multiobjective Flexible Job Shop by Deep Reinforcement Learning

Abstract: In modern discrete flexible manufacturing systems, dynamic disturbances frequently occur in real time and each job may contain several special operations in partial-no-wait constraint due to technological requirements. In this regard, a hierarchical multiagent deep reinforcement learning (DRL)-based real-time scheduling method named hierarchical multi-agent proximal policy optimization (HMAPPO) is developed to address the dynamic partial-no-wait multiobjective flexible job shop scheduling problem (DMOFJSP-PNW) with new job insertions and machine breakdowns. The proposed HMAPPO contains three proximal policy optimization (PPO)-based agents operating in different spatiotemporal scales, namely, objective agent, job agent, and machine agent. The objective agent acts as a higher controller periodically determining the temporary objectives to be optimized. The job agent and machine agent are lower actuators, respectively, choosing a job selection rule and machine assignment rule to achieve the temporary objective at each rescheduling point. Five job selection rules and six machine assignment rules are designed to select an uncompleted job and assign the next operation of which together with its successors in no-wait constraint on the corresponding processing machines. A hierarchical PPO-based training algorithm is developed. Extensive numerical experiments have confirmed the effectiveness and superiority of the proposed HMAPPO compared with other well-known dynamic scheduling methods. Note to Practitioners—The motivation of this article stems from the need to develop real-time scheduling methods for modern discrete flexible manufacturing factories, such as aerospace product manufacturing and steel manufacturing, where dynamic events frequently occur, and each job may contain several operations subjected to the no-wait constraint. Traditional dynamic scheduling methods, such as metaheuristics or dispatching rules, either suffer from poor time efficiency or fail to ensure good solution quality for multiple objectives in the long-term run. Meanwhile, few of the previous studies have considered the partial-no-wait constraint among several operations from the same job, which widely exists in many industries. In this article, we propose a hierarchical multiagent deep reinforcement learning (DRL)-based real-time scheduling method named HMAPPO to address the dynamic partial-no-wait multiobjective flexible job shop scheduling problem (DMOFJSP-PNW) with new job insertions and machine breakdowns. The proposed HMAPPO uses three DRL-based agents to adaptively select the temporary objectives and choose the most feasible dispatching rules to achieve them at different rescheduling points, through which the rescheduling can be made in real time and a good compromise among different objectives can be obtained in the long-term schedule. Extensive experimental results have demonstrated the effectiveness and superiority of the proposed HMAPPO. For industrial applications, this method can be extended to many other production scheduling problems, such as hybrid flow shops and open shop with different uncertainties and objectives.

A hybrid differential evolution algorithm for flexible job shop scheduling with outsourcing operations and job priority constraints

Abstract: • A flexible job shop scheduling problem with outsourcing and job priority is studied. • Resources that have different processing times to handle one operation are studied. • A sequence-based mathematical model which minimizes the overdue days is proposed. • A hybrid self-adaptive differential evolution algorithm (HSDE) is proposed. • Experiments are carried out to verify the effectiveness of the model and algorithm. Owing to the increasing complexity of products and the specialization of enterprises, production outsourcing has become a common practice in industrial manufacturing. Moreover, different jobs feature various priorities in actual production. The previous research that aims to minimize the makespan may not be applicable in real scenarios. Therefore, this study investigates a flexible job shop scheduling problem with outsourcing operations and job priority constraints. We propose a sequence-based mathematical model aiming at minimizing weighted overdue days, which considers the outsourcing constraints and different overdue weights of jobs with different priorities. An efficient hybrid self-adaptive differential evolution algorithm with heuristic strategies (HSDE) is proposed to address this problem. In HSDE, a well-designed chromosome encoding and decoding method is presented. To eliminate individuals that do not satisfy the outsourcing constraints, we add a penalty term to improve the objective function. By considering heuristic strategies for initial chromosome generation, the proposed approach is able to achieve a high-quality initial population. Crossover and mutation operators with self-adaptive control of the parameters are established to enlarge the search range and accelerate the convergence speed. Finally, several experiments are conducted to verify the effectiveness of the proposed model and algorithm. Experimental results confirm that the proposed algorithm outperforms other algorithms both in efficiency and accuracy.

Energy saving scheduling strategy for job shop under TOU and tiered electricity price

Abstract: In view of the energy-saving scheduling problem of job shop in the order-oriented manufacturing enterprise, it is necessary to consider the energy consumption of equipment processing and standby. Considering the Time-of-Use (TOU) and tiered electricity price, a mathematical model is established by introducing the switch strategy in the idle time of equipment to reduce the cost of power consumption in production and assembly. In order to solve the discontinuous domain problem that the standard Firefly Algorithm (FA) is not suitable for workshop energy-saving scheduling, an improved FA is used to solve the mathematical model. The coding, decoding and location updating of the algorithm are redesigned. A three-layer coding structure based on the workpiece sequence, starting time and switch strategy is designed; for the iterative process of workpiece sequence encoding, the Hamming distance is used to replace the Euclidean distance to calculate the position distance between the firefly individuals. Precedence preservation order-based crossover (POX) crossover is used to replace the position updating operation to ensure the feasibility of the solution. Finally, the correctness of the mathematical model and the effectiveness of the algorithm are verified by examples.

Robust scheduling for flexible machining job shop subject to machine breakdowns and new job arrivals considering system reusability and task recurrence

Abstract: This paper focuses on the production scheduling problem of flexible job shops. In the production process of flexible job shop, there are dynamic events such as machine breakdowns or new job arrivals, which will interfere with the implementation of scheduling scheme and reduce the stability of the system. In response to this problem, this paper proposes a new robust optimization method that considers dynamic events and designs two indicators for evaluating the robustness of the system, namely the reusability of the system and the reproducibility of processing tasks. Two indicators are used to evaluate the comprehensive reusability of the system jointly. In the process of system rescheduling, this paper proposes a dynamic event response strategy (DERS) considering the comprehensive reusability of the system and establishes a multi-objective optimization model considering the total energy consumption, the makespan, and the comprehensive reusability of the system. In order to solve the model efficiently and obtain the optimal Pareto frontier, a multi-objective particle swarm arithmetic optimization (PSAO) is proposed in this paper. Finally, this paper designs experiments based on standard data cases, solves them based on this model and compares them with other algorithms. The final results show that in the flexible job-shop scheduling process, this method can effectively adjust the scheduling plan to respond to dynamic events to achieve stable scheduling in an uncertain environment.

Multi-objective Q-learning-based hyper-heuristic with Bi-criteria selection for energy-aware mixed shop scheduling

Abstract: Owning to diverse customer demands and enormous product varieties, mixed shop production systems are applied in practice to improve responsiveness and productivity along with energy-saving. This work addresses a mixture of job-shop and flow-shop production scheduling problem with a speed-scaling policy and no-idle time strategy. A mixed-integer linear programming model is formulated to determine the speed level of operations and the sequence of job-shop and flow-shop products, aiming at the simultaneous optimization of production efficiency and energy consumption. Then, a multi-objective Q-learning-based hyper-heuristic with Bi-criteria selection (QHH-BS) is developed to obtain a set of high-quality Pareto frontier solutions. In this algorithm, a new three-layer encoding is designed to represent the production sequence of job-shop and flow-shop products; the Pareto-based and indicator-based selection criteria are sequentially implemented to encourage diversity and convergence; Q-learning with a multi-objective metric-based reward mechanism is applied to select an optimizer from three prominent optimizers in each iteration for better exploration and exploitation. Three conclusions are drawn from extensive experiments: (1) Bi-criteria selection is superior to single-criterion selections; (2) Q-learning-based hyper-heuristic is more effective and robust than single optimizer-based algorithms and simple hyper-heuristics; (3) QHH-BS outperforms the existing state-of-the-art multi-objective algorithms in convergence and diversity.

Digital twin data-driven proactive job-shop scheduling strategy towards asymmetric manufacturing execution decision

Abstract: The information asymmetry phenomenon widely exists in production management decisions due to the latency of manufacturing data transmissions. Also, stochastic events on the physical production site will result in information asymmetry, which may lead to inconsistency between current execution and previous resource allocation plans. It is meaningful and important for developing an information model based on the Internet of Manufacturing Things to timely and actively adjust the scheduling strategy to meet the symmetry requirements of the production execution process. Based on the digital twin data collected from the workshop, a proactive job-shop scheduling strategy was discussed in this paper. Firstly, the mechanism for the influence of delayed local operations on makespan was deduced. Then, a framework for implementing the proactive job-shop scheduling strategy was proposed. Coordination point was used to determine the adjustment interval of local operations; right-shift rule with delay time constraints was used to adjust the unprocessed operation sequences on machines. Finally, the examples including 6*6 (6 jobs, 6 machines) and 20*40 (20 jobs, 40 machines) were presented to verify the effectiveness and scalability of the proposed method. It can be predicted that the proactive scheduling strategy provides the online decisions for the efficient and smooth execution of the digital twin-driven workshop production.

Dynamic Scheduling Method for Job-Shop Manufacturing Systems by Deep Reinforcement Learning with Proximal Policy Optimization

Abstract: With the rapid development of Industrial 4.0, the modern manufacturing system has been experiencing profoundly digital transformation. The development of new technologies helps to improve the efficiency of production and the quality of products. However, for the increasingly complex production systems, operational decision making encounters more challenges in terms of having sustainable manufacturing to satisfy customers and markets’ rapidly changing demands. Nowadays, rule-based heuristic approaches are widely used for scheduling management in production systems, which, however, significantly depends on the expert domain knowledge. In this way, the efficiency of decision making could not be guaranteed nor meet the dynamic scheduling requirement in the job-shop manufacturing environment. In this study, we propose using deep reinforcement learning (DRL) methods to tackle the dynamic scheduling problem in the job-shop manufacturing system with unexpected machine failure. The proximal policy optimization (PPO) algorithm was used in the DRL framework to accelerate the learning process and improve performance. The proposed method was testified within a real-world dynamic production environment, and it performs better compared with the state-of-the-art methods.

Integrated scheduling of machines and automated guided vehicles (AGVs) in flexible job shop environment using genetic algorithms

Abstract: In this research integrated scheduling of machines and automated guided vehicles (AGVs) in a flexible job shop environment is addressed. The scheduling literature generally ignores the transportation of jobs between the machines and when considered typically assumes an unlimited number of AGVs. In order to comply with Industry 4.0 requirements, today’s manufacturing systems make use of AGVs to transport jobs between the machines. The addressed problem involves simultaneous assignment of operations to one of the alternative machines, determining the sequence of operations on each machine and assignment of transportation operations between machines to an available AGV. We present a Microsoft Excel® spreadsheet-based solution for the problem. Evolver®, a proprietary GA is used for the optimization. The GA routine works as an add-in to the spreadsheet environment. The flexible job shop model is developed in Microsoft Excel® spreadsheet. The assignment of AGV is independent of the GA routine and is done by the spreadsheet model while the GA finds the assignment of operations to the machines and then finds the best sequence of operations on each machine. Computational analysis demonstrates that the proposed method can effectively and efficiently solve a wide range of problems with reasonable accuracy. Benchmark problems from the literature are used to highlight the effectiveness and efficiency of the proposed implementation. In most of the cases the proposed implementation can find the best-known solution found by previous studies.

Digital twin data-driven proactive job-shop scheduling strategy towards asymmetric manufacturing execution decision

Abstract: The information asymmetry phenomenon widely exists in production management decisions due to the latency of manufacturing data transmissions. Also, stochastic events on the physical production site will result in information asymmetry, which may lead to inconsistency between current execution and previous resource allocation plans. It is meaningful and important for developing an information model based on the Internet of Manufacturing Things to timely and actively adjust the scheduling strategy to meet the symmetry requirements of the production execution process. Based on the digital twin data collected from the workshop, a proactive job-shop scheduling strategy was discussed in this paper. Firstly, the mechanism for the influence of delayed local operations on makespan was deduced. Then, a framework for implementing the proactive job-shop scheduling strategy was proposed. Coordination point was used to determine the adjustment interval of local operations; right-shift rule with delay time constraints was used to adjust the unprocessed operation sequences on machines. Finally, the examples including 6*6 (6 jobs, 6 machines) and 20*40 (20 jobs, 40 machines) were presented to verify the effectiveness and scalability of the proposed method. It can be predicted that the proactive scheduling strategy provides the online decisions for the efficient and smooth execution of the digital twin-driven workshop production.

Reinforcement learning for online optimization of job-shop scheduling in a smart manufacturing factory

Abstract: The job-shop scheduling problem (JSSP) is a complex combinatorial problem, especially in dynamic environments. Low-volume-high-mix orders contain various design specifications that bring a large number of uncertainties to manufacturing systems. Traditional scheduling methods are limited in handling diverse manufacturing resources in a dynamic environment. In recent years, artificial intelligence (AI) arouses the interests of researchers in solving dynamic scheduling problems. However, it is difficult to optimize the scheduling policies for online decision making while considering multiple objectives. Therefore, this paper proposes a smart scheduler to handle real-time jobs and unexpected events in smart manufacturing factories. New composite reward functions are formulated to improve the decision-making abilities and learning efficiency of the smart scheduler. Based on deep reinforcement learning (RL), the smart scheduler autonomously learns to schedule manufacturing resources in real time and improve its decision-making abilities dynamically. We evaluate and validate the proposed scheduling model with a series of experiments on a smart factory testbed. Experimental results show that the smart scheduler not only achieves good learning and scheduling performances by optimizing the composite reward functions, but also copes with unexpected events (e.g. urgent or simultaneous orders, machine failures) and balances between efficiency and profits.

An improved event-triggered predictive control for capacity adjustment in reconfigurable job-shops

Abstract: In order to regulate work in process (WIP) to the desired value in the job shop production control system, capacity adjustment as an effective and efficient measure, which is typically achieved by flexible staffs and working time. In this paper, instead of traditional labour-oriented approaches, we consider a machinery-based capacity adjustment via reconfigurable machine tools (RMTs) to compensate for unpredictable events. To this end, we employ model predictive control (MPC) in combination with genetic algorithm (GA) to explicitly consider complex reconfiguration strategies and address the related integer assignment optimisation problems. To further reduce energy consumption and avoid frequent and unnecessary reconfigurations while keeping a certain level of performance, we adopt an event-triggered MPC scheme with the proposed ‘Double-layer event-triggering conditions’. Through extensively illustrated simulations, we demonstrate the effectiveness and plug-and-play availability of the proposed method for a six-workstation four-product job shop system and compare it to a state-of-the-art method.

Differential Evolution Algorithm Combined with Uncertainty Handling Techniques for Stochastic Reentrant Job Shop Scheduling Problem

Abstract: This paper considers two kinds of stochastic reentrant job shop scheduling problems (SRJSSP), i.e., the SRJSSP with the maximum tardiness criterion and the SRJSSP with the makespan criterion. Owing to the NP-complete complexity of the considered RJSSPs, an effective differential evolutionary algorithm (DEA) combined with two uncertainty handling techniques, namely, DEA_UHT, is proposed to address these problems. Firstly, to reasonably control the computation cost, the optimal computing budget allocation technique (OCBAT) is applied for allocating limited computation budgets to assure reliable evaluation and identification for excellent solutions or individuals, and the hypothesis test technique (HTT) is added to execute a statistical comparison to reduce some unnecessary repeated evaluation. Secondly, a reentrant-largest-order-value rule is designed to convert the DEA’s individual (i.e., a continuous vector) to the SRJSSP’s solution (i.e., an operation permutation). Thirdly, a conventional active decoding scheme for the job shop scheduling problem is extended to decode the solution for obtaining the criterion value. Fourthly, an Insert-based exploitation strategy and an Interchange-based exploration strategy are devised to enhance DEA’s exploitation ability and exploration ability, respectively. Finally, the test results and comparisons manifest the effectiveness and robustness of the proposed DEA_UHT.

OUP accepted manuscript

Abstract: Abstract This paper addresses a variant of the job shop scheduling problem with total tardiness minimization where task durations and due dates are uncertain. This uncertainty is modelled with intervals. Different ranking methods for intervals are considered and embedded into a genetic algorithm. A new robustness measure is proposed to compare the different ranking methods and assess their capacity to predict ‘expected delays’ of jobs. Experimental results show that dealing with uncertainty during the optimization process yields more robust solutions. A sensitivity analysis also shows that the robustness of the solutions given by the solving method increases when the uncertainty grows.