Showing papers by "Ying Cheng published in 2020"

Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

Abstract: When watching videos, the occurrence of a visual event is often accompanied by an audio event, e.g., the voice of lip motion, the music of playing instruments. There is an underlying correlation between audio and visual events, which can be utilized as free supervised information to train a neural network by solving the pretext task of audio-visual synchronization. In this paper, we propose a novel self-supervised framework with co-attention mechanism to learn generic cross-modal representations from unlabelled videos in the wild, and further benefit downstream tasks. Specifically, we explore three different co-attention modules to focus on discriminative visual regions correlated to the sounds and introduce the interactions between them. Experiments show that our model achieves state-of-the-art performance on the pretext task while having fewer parameters compared with existing methods. To further evaluate the generalizability and transferability of our approach, we apply the pre-trained model on two downstream tasks, i.e., sound source localization and action recognition. Extensive experiments demonstrate that our model provides competitive results with other self-supervised methods, and also indicate that our approach can tackle the challenging scenes which contain multiple sound sources.

Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

Abstract: When watching videos, the occurrence of a visual event is often accompanied by an audio event, e.g., the voice of lip motion, the music of playing instruments. There is an underlying correlation between audio and visual events, which can be utilized as free supervised information to train a neural network by solving the pretext task of audio-visual synchronization. In this paper, we propose a novel self-supervised framework with co-attention mechanism to learn generic cross-modal representations from unlabelled videos in the wild, and further benefit downstream tasks. Specifically, we explore three different co-attention modules to focus on discriminative visual regions correlated to the sounds and introduce the interactions between them. Experiments show that our model achieves state-of-the-art performance on the pretext task while having fewer parameters compared with existing methods. To further evaluate the generalizability and transferability of our approach, we apply the pre-trained model on two downstream tasks, i.e., sound source localization and action recognition. Extensive experiments demonstrate that our model provides competitive results with other self-supervised methods, and also indicate that our approach can tackle the challenging scenes which contain multiple sound sources.

Hypernetwork-based manufacturing service scheduling for distributed and collaborative manufacturing operations towards smart manufacturing

Abstract: In the future smart manufacturing, both of sensor-based environment in shop floors and cloud-based environment among more and more enterprises are deployed gradually. Various distributed and separated manufacturing facilities are as collaborative cloud services, integrated and aggregated with their real-time information. It provides opportunities for the distributed and collaborative manufacturing operations across lots of distributed but networked enterprises on demand with enough flexibility. To this end, the scheduling problem and its result of those collaborative services for distributed manufacturing operations play an important role in improving manufacturing utilization and efficiency. In this paper, we put forward the hypernetwork-based models introducing the thought of graph coloring and an artificial bee colony algorithm based method for this scheduling problem. Three groups of experiments are carried out respectively to discuss therein different situations of distributed and collaborative manufacturing operations, i.e., in a private cloud, in a public cloud, and in a hybrid cloud. Some future studies with further consideration of collaboration equilibrium, dynamic control and data-based intelligence, are finally pointed out in the conclusion.

Manufacturing services collaboration: connotation, framework, key technologies, and research issues

Abstract: Manufacturing resources sharing and collaboration has become the inevitable trend to optimize the operation of manufacturing industry. The development of various advanced manufacturing systems (AMSs) and new-generated information technologies (New ITs) prompt the emergence of various industrial Internet platforms. In this context, it provides a brand new opportunity for enterprises to conduct manufacturing resources sharing and collaboration in the form of manufacturing services, namely, manufacturing services collaboration (MS-collaboration). However, the relevant research on MS-collaboration under the environment of industrial Internet platforms is still insufficient. Therefore, in this paper, the evolution from manufacturing collaboration to MS-collaboration toward industrial Internet platforms is analyzed at first, and the derived environment characteristics and challenges of MS-collaboration are discussed accordingly. After exploring the connotation of MS-collaboration, its implementation framework and corresponding enabling technologies are proposed. A 3D Printing case is presented to illustrate the application of the proposed methods. Finally, in order to promote the application of an industrial Internet platform, stimulate more enterprises to participate in, and conduct MS-collaboration in it, several future research issues are pointed out.

Scalable Hypernetwork-Based Manufacturing Services Supply Demand Matching Toward Industrial Internet Platforms

Abstract: With the deeper application of sensor & cloud-based environment into manufacturing, deploying the industrial Internet platforms toward smart manufacturing has been more concerned. Based on the platforms, ubiquitous enterprises could participate in and support cross-enterprise collaboration, so that their distributed manufacturing facilities and capabilities could be shared and utilized in the form of manufacturing services (MSs). However, in order to achieve the successful application of the platforms, how to settle the supply demand matching (SDM) of the distributed manufacturing facilities and capabilities in the form of MSs, namely, MSs-SDM, becomes one of the most urgent problems to be solved. In addition, the trend of manufacturing socialization makes this problem much more scalable. In this context, this article aims to establish a set of hypernetwork-based models for the scalable MSs-SDM problem at first. An enterprises collaborative network is derived which is the projection of the underlying MSs-SDM situation to the upper-layer enterprises. Second, a method according to the evaluation on the cross-enterprise collaboration is proposed for this problem. In which, the created utilities, the rates of service invocation, and task allocation from both the global view of the overall network and the local view of each participated enterprise are evaluated. Finally, two steps of experiments introducing scalabilities illustrate the feasibility of the proposed models and the effectiveness of the derived method for MSs-SDM optimization, and further reveal five managerial implications to improve the operation and industrial practice of the platforms.

Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication

Abstract: Visual storytelling aims to generate a narrative paragraph from a sequence of images automatically. Existing approaches construct text description independently for each image and roughly concatenate them as a story, which leads to the problem of generating semantically incoherent content. In this paper, we propose a new way for visual storytelling by introducing a topic description task to detect the global semantic context of an image stream. A story is then constructed with the guidance of the topic description. In order to combine the two generation tasks, we propose a multi-agent communication framework that regards the topic description generator and the story generator as two agents and learn them simultaneously via iterative updating mechanism. We validate our approach on VIST dataset, where quantitative results, ablations, and human evaluation demonstrate our method’s good ability in generating stories with higher quality compared to state-of-the-art methods.

An aggregated-tasks oriented manufacturing services scheduling toward industrial Internet platforms

Abstract: For pursuing smart manufacturing, the industrial Internet platforms have got more and more attention. An industrial Internet platform can gather almost all information flows, manufacturing capacities and production tools in the form of services to achieve sharing and collaboration of resources, subjects and knowledge. The aggregation characteristic of the platform results a problem, namely the distributive services scheduling for the concurrently submitted manufacturing tasks, which hinders the further application of an industrial Internet platform. In this paper, it aims to propose an aggregated-tasks oriented manufacturing services scheduling approach toward industrial Internet platforms for improving their effectiveness and global performance. Firstly, the dual-aggregation of manufacturing services and tasks in the platform is analyzed. Then based on aggregated manufacturing services collaboration, a services-aggregation aware modeling for the aggregated tasks is presented to describe the dual-aggregation characteristic and decompose the aggregated tasks. Finally, a manufacturing services scheduling optimization model is established and solved by particle swarm optimization algorithm. The experimental result proves the effectiveness of the proposed scheduling approach for the dual-aggregation of manufacturing services and tasks on the platform.

Efficient scheduling system for complex tasks of digital twin system

Abstract: The invention discloses an efficient scheduling system for complex tasks of a digital twin system The system is suitable for Virtex-5-series FPGA chips of the Xilinx company, and comprises the stepsof designing a digital twin system complex task attribute definition module, whereinthe module completes task attribute definition from the four aspects of task resource occupation, arrival time, execution time and deadline time; designing a complex task scheduling double-cache module of the digital twin system, and finishing caching of tasks arriving in batches by the module so as to calculate scheduling priorities of the tasks at the same moment, and finishing re-scheduling of tasks failed in previous scheduling under the constraint of a certain scheduling frequency; and designing a digitaltwin system complex task schedulability judgment module, completing task scheduling priority calculation by the module, and performing task scheduling according to the scheduling priority The problemthat the emergency task cannot be immediately scheduled can be solved to a certain extent, and the complex task scheduling efficiency of the digital twin system is improved

Efficient division method for complex tasks of digital twin system

Abstract: The invention discloses an efficient division method for the complex tasks of a digital twin system. The method is suitable for a heterogeneous multi-core SoC environment of the Xilinx ZYNQ-7000 SoC design, and comprises the steps of designing a complex task division early-stage module of the digital twin system, wherein the module solves the minimum task overall execution time based on a classical genetic algorithm; designing a complex task division later-stage module of the digital twin system, wherein the module solves the minimum task overall execution time based on a greedy algorithm; designing a complex task division scheduling module 1 of the digital twin system, wherein the module completes the stop of the early-stage module and the start of the later-stage module; and designing acomplex task division scheduling module 2 of the digital twin system, wherein the module determines whether to perform task division iteration again or not by monitoring the number of iterations of the later-stage module, and calculates and outputs an optimal division scheme. According to the method, the classic genetic algorithm and the greedy algorithm are combined, so that the global and localoptimization capability of the task division is ensured, and the complex task division efficiency of the digital twin system is improved.