To facilitate the personalized smart manufacturing paradigm with cognitive automation capabilities, Deep Reinforcement Learning (DRL) has attracted ever-increasing attention by offering an adaptive and flexible solution. DRL takes the advantages of both Deep Neural Networks (DNN) and Reinforcement Learning (RL), by embracing the power of representation learning, to make precise and fast decisions when facing dynamic and complex situations. Ever since the first paper of DRL was published in 2013, its applications have sprung up across the manufacturing field with exponential publication growth year by year. However, there still lacks any comprehensive review of the DRL in the field of smart manufacturing. To fill this gap, a systematic review process was conducted, with 261 relevant publications selected to date (20-Oct-2022), to gain a holistic understanding of the development, application, and challenges of DRL in smart manufacturing along the whole engineering lifecycle. First, the concept and development of DRL are summarized. Then, the typical DRL applications are analyzed in the four engineering lifecycle stages: design, manufacturing, distribution, and maintenance. Finally, the challenges and future directions are illustrated, especially emerging DRL-related technologies and solutions that can improve the manufacturing system’s deployment feasibility, cognitive capability, and learning efficiency, respectively. It is expected that this work can provide an insightful guide to the research of DRL in the smart manufacturing field and shed light on its future perspectives. 

Deep reinforcement learning in smart manufacturing: A review and prospects

The demand for individualized products drives modern manufacturing systems towards greater adaptability and flexibility. This increases the focus on data-driven digital twins enabling swift adaptations. Within the framework of cyber-physical systems, the digital twin is a digital model that is fully connected to the physical and digital assets. A digital model must follow a standardization for interoperable data exchange. Established ontologies and meta-models offer a basis in the definition of a schema, which is the first phase of creating a digital twin. The next phase is the standardized and structured modeling with static use-case specific data. The final phase is the deployment of digital twins into operation with a full connection of the digital model with the remaining cyber-physical system. In this deployment phase communication standards and protocols provide a standardized data exchange. A survey on the state-of-the-art of these three digital twin phases reveals the lack of a consistent workflow from ontology-driven definition to standardized modeling. Therefore, one goal of this paper is the design of an end-to-end digital twin pipeline to lower the threshold of creating and deploying digital twins. As the task of establishing a communication connection is highly repetitive, an automation concept by providing structured protocol data is the second goal. The planning and control of a line-less assembly system with manual stations and a mobile robot as resources and an industrial dog as the product serve as exemplary digital twin applications. Along this use-case the digital twin pipeline is transparently explained.

/pdf/pipeline-for-ontology-based-modeling-and-automated-14n55n8s9b.pdf

Pipeline for ontology-based modeling and automated deployment of digital twins for planning and control of manufacturing systems

This paper describes a low-power low-cost 24-bit Σ-digital-to-analog converter (DAC) for portable digital-audio applications.The interpolation filter uses a no-multiplier scheme to implement the arithmetic units and reading-writing common storage scheme for the delay-line to significantly reduce the die area.A 15-level quantizer,third-order,single-stage Σ-modulator is employed to reduce the passband quantization noise,relax the out-of-band filtering requirements,and enhance immunity to clock jitter.A data weighted averaging algorithm is used to mitigate the nonlinearity caused by capacitor mismatch.A direct charge transfer switched-capacitor low-pass filter (DCT-SC LPF) is used to reconstruct the analog signal to reduce the kT/C noise and capacitor mismatch effect with a small increase of the power dissipation.The chip was fabricated in the SMIC 0.13 μm 1P5M CMOS process.The cell area of the digital part is 0.056 mm 2 and the total area of the analog part is 0.34 mm 2.The supply voltage is 1.2 V for the digital circuit and 3.3 V for the analog circuit.The power consumption of the analog part is 3.5 mW.The audio DAC achieves a 100 dB dynamic range and an 84 dB peak signal-to-noise-plus-distortion ratio over a 20 kHz passband.The results show that these performances are good enough for high quality portable audio applications.

24-bit Low-Power Low-Cost Digital Audio Sigma-Delta DAC

Deep Reinforcement Learning for autonomous pre-failure tool life improvement

A digital shadow framework using distributed system concepts

Digital twins represent physical systems through dynamic adaptive digital replicas. These replicas are virtual images of the functionality and interactions of the physical system and its components. Digital twin provides real-time monitoring and decision-making support. These are essential pillars in the Industry 4.0 paradigm. On the system level, multiple architectures are established for digital twin concepts in the literature. However, the roadmap for deploying a functional digital twin has not been fully recognized thus far. In this paper, we propose a framework for the deployment of a digital twin in production systems. The framework covers both levels of virtualization; digital shadowing, and digital twining. It utilizes present technologies in network communication, data management, and knowledge extraction. We describe the main components of the digital twin, and their relation to the management schemes currently implemented in production systems. Additionally, we devise an approach for integration of available simulation and data analytic tools for dynamic modeling and system performance evaluation. Through our proposed framework, the current technologies and tools are capable of deploying a digital twin. Consequently, learning algorithms and monitoring procedures are exploited for dynamic performance improvement.

A Framework for Digital Twin Deployment in Production Systems

Fabrication areas in semiconductor industry are considered one of the most complex production systems. This complexity is caused by the high-mix of products and end-user market-based demands in that industry. Its dynamic and challenging processing requirements affect the handling capabilities of traditional production management paradigms. In this paper, we propose an application for dispatching and resources allocation through reinforcement learning. The application is based on a discrete-event simulation model for a case study of a real semiconductor manufacturing system. The model is built using both data-driven and agent-based approaches. The model simulates the various processing aspects that are present normally in these complex systems. The model’s agents are responsible for dispatching tasks and allocation of the different system’s resources. They employ Deep-Q-Network reinforcement learning. They learn simultaneously through the model execution. An independent Deep-Q-Network is trained for each agent. The model provides the training environment for the agents in which their decisions are applied and assessed for their adequacy. Our formulation of the environment’s state and the reward function for the learning algorithms creates cooperative decision-making policies for the agents. This results in improving the global performance of the whole system, and the performance of each agent’s resources. Our approach is compared to heuristics-based strategies that are applied in our case study. It achieved better production performance than the currently applied strategy.

Simulation and deep reinforcement learning for adaptive dispatching in semiconductor manufacturing systems

Modern digital systems require one or more frequency shaping modules that realize certain transfer functions, usually in the form of digital filters. Breaking up the design into multiple stages results in reducing the implementation area and power consumption. However, there is no well-defined automation methodology to design an optimum multistage multirate filter. This article proposes a new design flow automation tool using a hybrid sampling strategy-based differential evolution (HSS-DE). The user specifies the inputs and outputs of the system, including the input spectrum, the sample rate change, the magnitude and phase responses, and the required signal-to-noise ratio. The tool uses this information to generate a solution indicating the number of stages and the filters’ specifications for each stage such that the generated architecture satisfies the user requirements. We compared the proposed tool to subsystems designed by experts in the ASIC industry. Several case studies are presented to demonstrate the proposed tool’s efficiency in different design problems. The case studies include decimation filters for sigma–delta ( $\Sigma \Delta $ ) analog-to-digital converters (ADCs) and interpolation filters for wideband transmitters such as wideband code division multiple access (WCDMA). The case studies’ results indicate that the proposed design automation flow reduces the design time remarkably, in addition to reducing the implementation in all cases.

Ayman AboElHassan

Papers

A Framework for Digital Twin Deployment in Production Systems

Simulation and deep reinforcement learning for adaptive dispatching in semiconductor manufacturing systems

Multistage Multirate Transfer Function Automated Synthesis Using Hybrid Sampling Strategy-Based Differential Evolution