Showing papers by "Chang Liu published in 2022"

Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets

Abstract: This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. With these simple modifications, Graphormer could attain better results on large-scale molecular modeling datasets than the vanilla one, and the performance gain could be consistently obtained on 2D and 3D molecular graph modeling tasks. In addition, we show that with a global receptive field and an adaptive aggregation strategy, Graphormer is more powerful than classic message-passing-based GNNs. Empirically, Graphormer could achieve much less MAE than the originally reported results on the PCQM4M quantum chemistry dataset used in KDD Cup 2021. In the meanwhile, it greatly outperforms the competitors in the recent Open Catalyst Challenge, which is a competition track on NeurIPS 2021 workshop, and aims to model the catalyst-adsorbate reaction system with advanced AI models. All codes could be found at https://github.com/Microsoft/Graphormer.

Direct Molecular Conformation Generation

Abstract: Molecular conformation generation aims to generate three-dimensional coordinates of all the atoms in a molecule and is an important task in bioinformatics and pharmacology. Previous methods usually first predict the interatomic distances, the gradients of interatomic distances or the local structures (e.g., torsion angles) of a molecule, and then reconstruct its 3D conformation. How to directly generate the conformation without the above intermediate values is not fully explored. In this work, we propose a method that directly predicts the coordinates of atoms: (1) the loss function is invariant to roto-translation of coordinates and permutation of symmetric atoms; (2) the newly proposed model adaptively aggregates the bond and atom information and iteratively refines the coordinates of the generated conformation. Our method achieves the best results on GEOM-QM9 and GEOM-Drugs datasets. Further analysis shows that our generated conformations have closer properties (e.g., HOMO-LUMO gap) with the groundtruth conformations. In addition, our method improves molecular docking by providing better initial conformations. All the results demonstrate the effectiveness of our method and the great potential of the direct approach. The code is released at https://github.com/DirectMolecularConfGen/DMCG

Multi-Granularity Structural Knowledge Distillation for Language Model Compression

Abstract: Transferring the knowledge to a small model through distillation has raised great interest in recent years. Prevailing methods transfer the knowledge derived from mono-granularity language units (e.g., token-level or sample-level), which is not enough to represent the rich semantics of a text and may lose some vital knowledge. Besides, these methods form the knowledge as individual representations or their simple dependencies, neglecting abundant structural relations among intermediate representations. To overcome the problems, we present a novel knowledge distillation framework that gathers intermediate representations from multiple semantic granularities (e.g., tokens, spans and samples) and forms the knowledge as more sophisticated structural relations specified as the pair-wise interactions and the triplet-wise geometric angles based on multi-granularity representations. Moreover, we propose distilling the well-organized multi-granularity structural knowledge to the student hierarchically across layers. Experimental results on GLUE benchmark demonstrate that our method outperforms advanced distillation methods.

Anomalous ferroelectricity and double-negative effects in bilayer hexagonal boron nitride

Abstract: Recently, the presence of an out-of-plane ferroelectric polarization was experimentally confirmed in bilayer two-dimensional (2D) hexagonal boron nitride (h-BN). In this work, we employ first-principles computations to reveal that (i) such slidetronic ferroelectricity is electronic in nature and arises from in-plane interlayer translation, which is distinctive among 2D slidetronic ferroelectric materials; and (ii) there is a coexistence of a negative longitudinal piezoelectric effect and an out-of-plane negative Poisson's ratio in the ferroelectric state of bilayer h-BN, which we refer to as anomalous double-negative effects.

ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation

Abstract: Typical generative dialogue models utilize the dialogue history to generate the response. However, since one dialogue utterance can often be appropriately answered by multiple distinct responses, generating a desired response solely based on the historical information is not easy. Intuitively, if the chatbot can foresee in advance what the user would talk about (i.e., the dialogue future) after receiving its response, it could possibly provide a more informative response. Accordingly, we propose a novel dialogue generation framework named ProphetChat that utilizes the simulated dialogue futures in the inference phase to enhance response generation. To enable the chatbot to foresee the dialogue future, we design a beam-search-like roll-out strategy for dialogue future simulation using a typical dialogue generation model and a dialogue selector. With the simulated futures, we then utilize the ensemble of a history-to-response generator and a future-to-response generator to jointly generate a more informative response. Experiments on two popular open-domain dialogue datasets demonstrate that ProphetChat can generate better responses over strong baselines, which validates the advantages of incorporating the simulated dialogue futures.

An Improved MobileNet Network with Wavelet Energy and Global Average Pooling for Rotating Machinery Fault Diagnosis

Abstract: In recent years, neural networks have shown good performance in terms of accuracy and efficiency. However, along with the continuous improvement in diagnostic accuracy, the number of parameters in the network is increasing and the models can often only be run in servers with high computing power. Embedded devices are widely used in on-site monitoring and fault diagnosis. However, due to the limitation of hardware resources, it is difficult to effectively deploy complex models trained by deep learning, which limits the application of deep learning methods in engineering practice. To address this problem, this article carries out research on network lightweight and performance optimization based on the MobileNet network. The network structure is modified to make it directly suitable for one-dimensional signal processing. The wavelet convolution is introduced into the convolution structure to enhance the feature extraction ability and robustness of the model. The excessive number of network parameters is a challenge for the deployment of networks and also for the running performance problems. This article analyzes the influence of the full connection layer size on the total network. A network parameter reduction method is proposed based on GAP to reduce the network parameters. Experiments on gears and bearings show that the proposed method can achieve more than 97% classification accuracy under the strong noise interference of −6 dB, showing good anti-noise performance. In terms of performance, the network proposed in this article has only one-tenth of the number of parameters and one-third of the running time of standard networks. The method proposed in this article provides a good reference for the deployment of deep learning intelligent diagnosis methods in embedded node systems.

Invertible Rescaling Network and Its Extensions

Abstract: Image rescaling is a commonly used bidirectional operation, which first downscales high-resolution images to fit various display screens or to be storage- and bandwidth-friendly, and afterward upscales the corresponding low-resolution images to recover the original resolution or the details in the zoom-in images. However, the non-injective downscaling mapping discards high-frequency contents, leading to the ill-posed problem for the inverse restoration task. This can be abstracted as a general image degradation–restoration problem with information loss. In this work, we propose a novel invertible framework to handle this general problem, which models the bidirectional degradation and restoration from a new perspective, i.e. invertible bijective transformation. The invertibility enables the framework to model the information loss of pre-degradation in the form of distribution, which could mitigate the ill-posed problem during post-restoration. To be specific, we develop invertible models to generate valid degraded images and meanwhile transform the distribution of lost contents to the fixed distribution of a latent variable during the forward degradation. Then restoration is made tractable by applying the inverse transformation on the generated degraded image together with a randomly-drawn latent variable. We start from image rescaling and instantiate the model as Invertible Rescaling Network, which can be easily extended to the similar decolorization–colorization task. We further propose to combine the invertible framework with existing degradation methods such as image compression for wider applications. Experimental results demonstrate the significant improvement of our model over existing methods in terms of both quantitative and qualitative evaluations of upscaling and colorizing reconstruction from downscaled and decolorized images, and rate-distortion of image compression. Code is available at https://github.com/pkuxmq/Invertible-Image-Rescaling .

Assessment of GF3 Full-Polarimetric SAR Data for Dryland Crop Classification with Different Polarimetric Decomposition Methods

Abstract: Crop classification is one of the most important agricultural applications of remote sensing. Many studies have investigated crop classification using SAR data, while few studies have focused on the classification of dryland crops by the new Gaofen-3 (GF3) SAR data. In this paper, taking Hengshui city as the study area, the performance of the Freeman–Durden, Sato4, Singh4 and multi-component decomposition methods for dryland crop type classification applications are evaluated, and the potential of full-polarimetric GF3 data in dryland crop type classification are also investigated. The results show that the multi-component decomposition method produces the most accurate overall classifications (88.37%). Compared with the typical polarization decomposition techniques, the accuracy of the classification results using the new decomposition method is improved. In addition, the Freeman method generally yields the third-most accurate results, and the Sato4 (87.40%) and Singh4 (87.34%) methods yield secondary results. The overall classification accuracy of the GF3 data is very positive. These results demonstrate the great promising potential of GF3 SAR data for dryland crop monitoring applications.

Reciprocal Learning of Knowledge Retriever and Response Ranker for Knowledge-Grounded Conversations

Abstract: Grounding dialogue agents with knowledge documents has sparked increased attention in both academia and industry. Recently, a growing body of work is trying to build retrieval-based knowledge-grounded dialogue systems. While promising, these approaches require collecting pairs of dialogue context and the corresponding ground-truth knowledge sentences that contain the information regarding the dialogue context. Unfortunately, hand-labeling data to that end is time-consuming, and many datasets and applications lack such knowledge annotations. In this paper, we propose a reciprocal learning approach to jointly optimize a knowledge retriever and a response ranker for knowledge-grounded response retrieval without ground-truth knowledge labels. Specifically, the knowledge retriever uses the feedback from the response ranker as pseudo supervised signals of knowledge retrieval for updating its parameters, while the response ranker also receives the top-ranked knowledge sentences from knowledge retriever for optimization. Evaluation results on two public benchmarks show that our model can significantly outperform previous state-of-the-art methods.

Spatial Input–Output Analysis of Actuated Turbulent Boundary Layers

Abstract: This paper develops a spatial input–output approach to investigate the dynamics of a turbulent boundary layer subject to a localized single frequency excitation. One-way spatial integration is used to reformulate the problem in terms of spatial evolution equations. The associated input-output operator is then used to examine the effect of localized periodic actuation at a given temporal frequency, based on an experimental setup in which an active large scale structure is introduced into the outer layer of a turbulent boundary layer. First, the large-scale structures associated with the phase-locked modal velocity field obtained from spatial input–output analysis are shown to closely match those computed based on hot-wire measurements. The approach is then used to further investigate the response of the boundary layer to the synthetically generated large-scale structures. A quadrant trajectory analysis indicates that the spatial input–output response produces shear stress distributions consistent with those in canonical wall-bounded turbulent flows in terms of both the order and types of events observed. The expected correspondence between the dominance of different quadrant behavior and actuation frequency is also observed. These results highlight the promise of a spatial input–output framework for analyzing the formation and streamwise evolution of structures in actuated wall-bounded turbulent flows.

An Improved Convolutional Capsule Network for Compound Fault Diagnosis of RV Reducers

Abstract: In fault diagnosis research, compound faults are often regarded as an isolated fault mode, while the association between compound faults and single faults is ignored, resulting in the inability to make accurate and effective diagnoses of compound faults in the absence of compound fault training data. In an examination of the rotate vector (RV) reducer, a core component of industrial robots, this paper proposes a compound fault identification method that is based on an improved convolutional capsule network for compound fault diagnosis of RV reducers. First, one-dimensional convolutional neural networks are used as feature learners to deeply mine the feature information of a single fault from a one-dimensional time-domain signal. Then, a capsule network with a two-layer stack structure is designed and a dynamic routing algorithm is used to decouple and identify the single fault characteristics for compound faults to undertake the diagnosis of compound faults of RV reducers. The proposed method is verified on the RV reducer fault simulation experimental bench, the experimental results show that the method can not only diagnose a single fault, but it is also possible to diagnose the compound fault that is composed of two types of single faults through the learning of two types of single faults of the RV reducer when the training data of the compound faults of the RV reducer are missing. At the same time, the proposed method is used for compound fault diagnosis of bearings, and the experimental results confirm its applicability.

Adam: Dense Retrieval Distillation with Adaptive Dark Examples

Abstract: To improve the performance of the dual-encoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works construct the candidate passages following the supervised learning setting where a query is paired with a positive passage and a batch of negatives. However, through empirical observation, we find that even the hard negatives from advanced methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query through mixing-up and masking in discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher's confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.

Received Signal Strength-Based Wireless Source Localization With Inaccurate Anchor Positions

Abstract: Received signal strength (RSS)-based wireless localization is easy to implement at low cost. In practice, exact positions of anchors may not be available. This article focuses on determining the location of a source in the presence of inaccurate positions of anchors based on RSS directly. We first use Taylor expansion and a min–max approach to get a maximum likelihood estimator of the coordinates of the source. Then we propose a relaxed semidefinite programming model to circumvent the nonconvexity. We also propose a rounding algorithm considering both inaccurate source locations and inaccurate anchor locations. Simulation results together with analysis are presented to validate the proposed method.

An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets

Abstract: This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. The"Graphormer-V2"could attain better results on large-scale molecular modeling datasets than the vanilla one, and the performance gain could be consistently obtained on downstream tasks. In addition, we show that with a global receptive field and an adaptive aggregation strategy, Graphormer is more powerful than classic message-passing-based GNNs. Graphormer-V2 achieves much less MAE than the vanilla Graphormer on the PCQM4M quantum chemistry dataset used in KDD Cup 2021, where the latter one won the first place in this competition. In the meanwhile, Graphormer-V2 greatly outperforms the competitors in the recent Open Catalyst Challenge, which is a competition track on NeurIPS 2021 workshop, and aims to model the catalyst-adsorbate reaction system with advanced AI models. All models could be found at \url{https://github.com/Microsoft/Graphormer}.

Rethinking Task-Specific Knowledge Distillation: Contextualized Corpus as Better Textbook

Abstract: Knowledge distillation has been proven effective when customizing small language models for specific tasks. Here, a corpus as ‘textbook’ plays an indispensable role, only through which the teacher can teach the student. Prevailing methods adopt a two-stage distillation paradigm: general distillation first with task-agnostic general corpus and task-specific distillation next with augmented task-specific corpus. We argue that such a paradigm may not be optimal. In general distillation, it’s extravagant to let the diverse but desultory general knowledge overwhelms the limited model capacity of the student. While in task-specific distillation, the task corpus is usually limited and narrow, preventing the student from learning enough knowledge. To mitigate the issues in the two gapped corpora, we present a better textbook for the student to learn: contextualized corpus that contextualizes task corpus with large-scale general corpus through relevance-based text retrieval. Experimental results on GLUE benchmark demonstrate that contextualized corpus is the better textbook compared with jointly using general corpus and augmented task-specific corpus. Surprisingly, it enables task-specific distillation from scratch without general distillation while maintaining comparable performance, making it more flexible to customize the student model with desired model size under various computation constraints.

Fuzzy Logic Based Agent Selection for Failure Management in VANETs

Abstract: In VANETs, due to software or hardware mal-function, a failure node may absorb the network traffic and drop all packets. Failure management (i.e. identifying the failure node), plays a vital role in this situation. This task is performed by selected vehicles in the network, which are called 'agents'. Improper choice of agents can degrade the final failure node detection performance severely. For distributed schemes, agent selection for VANETs is challenging because of quick topology changes. In this paper, a fuzzy logic based agent selection method is proposed. This method comprehensively considers velocity, leadership, and routes of candidate vehicles. When the appropriate agent is selected, we deploy a neural network based failure node detection algorithm on it. We use numerical simulations to validate the effectiveness of the proposed method.

Self-FTS: A Self-Supervised Learning Method for Financial Time Series Representation in Stock Intraday Trading

Abstract: The stock price’s highly unstable fluctuation pattern makes learning efficient representation challenging to model the stock movement. The common deep learning often overfits after a few epochs of training and performs poorly in the validation set because the optimization objective is insufficient to characterize the stock adequately. In this paper, we propose Self-FTS, a self-supervised learning framework for financial time series representation, to learn the underlying representation and use in stock trading, affected by the fact that self-supervised learning is a promising technique for learning representation for extracting high dimensional features from unlabeled financial data to overcome the bias caused by handcrafted features. Specifically, we design several auxiliary tasks to generate samples with pseudo labels from the A-share stock price data sets and build a weight-sharing feature extraction backbone combined with a classification head to learn the pseudo labels based on the samples. Finally, We evaluate the learned representations extracted from the backbone by fine-tuning data sets labelled with stock returns to build an investment portfolio. Experimental analysis results on the Chinese stock market data show that our method significantly improves the stock trend forecasting performances and the actual investment income through backtesting compared to the current SOTA method, which strongly demonstrates our effective approach.

Hard Disk Drive Failure Prediction Based on GRU Neural Network

Abstract: Hard disk drive (HDD) failure prediction is critical for data center maintenance. The conventional long short-term memory (LSTM) neural networks have been successfully used to predict the failure of HDD. However, the short degradation cycle of the disk leads to a severe imbalance in the ratio of healthy data to failure data, which degrades the performance of LSTM neural networks. Moreover, the complexity of LSTM neural networks is also very high. This paper proposes a disk failure prediction approach based on an intelligent attribute gated recurrent unit (GRU) neural network and TimeGAN adversarial network, the GRU neural network can adapt to the impact of long hard disk data sequences, while the TimeGAN can address the data imbalance problem. Experiments performed on Backblaze data demonstrate that our proposed approach achieves better performances in terms of failure detection rate than then conventional LSTM.

Ball Screw Fault Diagnosis Based on Wavelet Convolution Transfer Learning

Abstract: The ball screw is the core component of the CNC machine tool feed system, and its health plays an important role in the feed system and even in the entire CNC machine tool. This paper studies the fault diagnosis and health assessment of ball screws. Aiming at the problem that the ball screw signal is weak and susceptible to interference, using a wavelet convolution structure to improve the network can improve the mining ability of signal time domain and frequency domain features; aiming at the challenge of ball screw sensor installation position limitation, a transfer learning method is proposed, which adopts the domain adaptation method as jointly distributed adaptation (JDA), and realizes the transfer diagnosis across measurement positions by extracting the diagnosis knowledge of different positions of the ball screw. In this paper, the adaptive batch normalization algorithm (AdaBN) is introduced to enhance the proposed model so as to improve the accuracy of migration diagnosis. Experiments were carried out using a self-made lead screw fatigue test bench. Through experimental verification, the method proposed in this paper can extract effective fault diagnosis knowledge. By collecting data under different working conditions at the bearing seat of the ball screw, the fault diagnosis knowledge is extracted and used to identify and diagnose the position fault of the nut seat. In this paper, some background noise is added to the collected data to test the robustness of the proposed network model.

Investigation of Statin Medication Use in Elderly Patients with Cardiovascular Disease on Regular Physical Examination and the Relationship with Glucolipid Metabolism and Adverse Cardiovascular Prognosis

Abstract: Our purpose of this study was to investigate the use of statins in elderly patients with cardiovascular diseases during regular physical examination and to analyze the relationship between statins and glucose and lipid metabolism and adverse cardiovascular prognosis. From January 2019 to December 2021, 2121 elderly patients with cardiovascular disease underwent regular physical examination as the study subjects to investigate the use and intensity of statins. The patients were divided into the dosing group (n = 1848) and the nondosing group (n = 273) according to whether they were taking statins or not. The cardiac function, glucose and lipid metabolism indexes, and cardiovascular adverse events were compared between the two groups. Statin use in elderly patients with cardiovascular disease was 87.13% (1848/2121). The intensity of statin use decreased with age (P < 0.05); the left ventricular ejection fraction (LVEF) was greater in the medicated group than in the nonmedicated group, and the left ventricular end-diastolic internal diameter (LVDd) and left ventricular end-systolic internal diameter (LVDs) were smaller than in the nonmedicated group (P < 0.05). The total cholesterol (TC), triglyceride (TG), low-density lipoprotein cholesterol (LDL-C), and fasting blood glucose (FBG) levels were lower in the medicated group than in the nonmedicated group, the high-density lipoprotein cholesterol (HDL-C) levels were higher than in the nonmedicated group, and the glycated hemoglobin (HbA1c) values were lower than in the nonmedicated group (P < 0.05). The overall incidence of cardiovascular adverse events in the medicated group was lower than that in the nonmedicated group (P < 0.05). Statin use was higher in elderly patients with cardiovascular disease; the intensity of drug use decreased with age. The patients' cardiac function, glucose metabolism, and prognosis were significantly improved after statin treatment.

Multisource Image Matching Method Using Hierarchical Structure Constraint and Phase Congruency

Abstract: Multisource image matching is still a challenging task due to the significant nonlinear radiometric differences and scale variations. To address the problem, we present a novel phase congruency approach and hierarchical structure constraint strategy for multisource image matching. Specifically, we employ the phase congruency of image frequency domain in Gaussian scale space, and KAZE operator was used to detect feature point in the maximum moment space, which was obtained by the Fourier transform of the Log-Gabor even-symmetric filter. And then, extended phase features within the neighborhood region were generated, next feature description in the framework of polar coordinates was obtained. Finally, in the stage of image matching, we use the hierarchical structure constraint strategy for random sampling and verification. The experiments show that the proposed algorithm is superior to d2-NET, LGHD, RIFT, and other mainstream multisource image matching methods. Especially for scale change and rotation, the proposed algorithm provides a better way to describe the common features of multisource images, and results in a reliable and accurate matching for multisource image with obvious intensity and radiation differences.

SOC Estimation of Lithium Batteries Based on Cluster-Growing Leaky Intergrator Echo State Network

Abstract: This thesis provides a new estimation model called the Cluster-Growing Leaky Integrator Echo State Network (CGLESN) to improve the accuracy of traditional neural networks for Lithium batteries SOC estimation and speed up the convergence. CGLESN changes the traditional Leaky-ESN with fixed reservoir size generated with experience. CGLESN enabled the reservoir to grow clustered according to the task requirements and determined the model structure when mission deadlines are met. CGLESN improved the structure of completely random generation of reservoir in traditional echo state network, which can better adapt to various tasks. The collected voltage, current and temperature under DST conditions are used as the model input, and SOC is used as the output. CGLESN is used to train and test the collected data as the estimation model. The results show that the CGLESN has higher prediction accuracy compared to traditional network models, such as: BP networks and standard Leaky-ESN networks.

Cooperative Edge Caching for Vehicular Networks in Road Intersections

Abstract: Cooperative edge caching in vehicular ad hoc net-works is expected to support massive data transmissions with low transmission delays, which is required in various powerful applications brought by the proliferation of smart vehicles and rapid development of B5G/6G communication techniques. How-ever, due to highly dynamic topology of vehicular networks and constrained storage capacities of edge servers, the design of effective caching scheme is challenging, especially in road intersections because of its complexity and flexibility. In this paper, we propose a cooperative edge caching scheme in a four-way intersection of two-lane roads to reduce the average download delay of content, which allows vehicles to fetch contents from road side units (RSUs) or base stations (BSs). To characterize the variation of communication rates between moving vehicles and RSUs, we divide coverage areas of RSUs into different communication zones, based on which we mathematically derived the average download delay of requested contents. Then we formulated an optimization problem of cooperative content placement to minimize the overall download delay, and proposed a dynamic programming (DP)-based algorithm to solve this nonlinear integer programming (NLIP) problem. Simulation results show that the proposed DP-based cooperative caching placement scheme is more efficient for average download delay reduction compared with benchmark schemes such as a popularity-based caching placement scheme and a random caching placement scheme.