Data Mining Practical Machine Learning Tools and Techniques

Matrix Factorization Techniques for Recommender Systems

本文对国际科学计量学杂志《Ｓｃｉｅｎｔｏｍｅｔｒｉｃｓ》１９７９－１９９１年的研究论文内容、栏目、作者及国别和编委及国别作了计量分析，揭示出科学计量学研究的重点、活动的中心及发展趋势，说明了学科带头人在发展科学计量学这门新兴学科中的作用。

国际刊物《Scientometrics》文献计量研究

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

Big Bird: Transformers for Longer Sequences

Traditional and recent approaches in background modeling for foreground detection: An overview

It has recently been shown that only a small number of samples from a low-rank matrix are necessary to reconstruct the entire matrix. We bring this to bear on computer vision problems that utilize low-dimensional subspaces, demonstrating that subsampling can improve computation speed while still allowing for accurate subspace learning. We present GRASTA, Grassmannian Robust Adaptive Subspace Tracking Algorithm, an online algorithm for robust subspace estimation from randomly subsampled data. We consider the specific application of background and foreground separation in video, and we assess GRASTA on separation accuracy and computation time. In one benchmark video example [16], GRASTA achieves a separation rate of 46.3 frames per second, even when run in MATLAB on a personal laptop.

Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video

This paper presents GRASTA (Grassmannian Robust Adaptive Subspace Tracking Algorithm), an ecient and robust online algorithm for tracking subspaces from highly incomplete information. The algorithm uses a robust l 1 -norm cost function in order to estimate and track non-stationary subspaces when the streaming data vectors are corrupted with outliers. We apply GRASTA to the problems of robust matrix completion and real-time separation of background from foreground in video. In this second application, we show that GRASTA performs highquality separation of moving objects from background at exceptional speeds: In one popular benchmark video example [28], GRASTA achieves a rate of 57 frames per second, even when run in MATLAB on a personal laptop.

Online Robust Subspace Tracking from Partial Information

In recommender systems, many efforts have been made on utilizing textual information in matrix factorization to alleviate the problem of data sparsity. Recently, some of the works have explored neural networks to do an in-depth understanding of textual item content and achieved impressive effectiveness by generating more accurate item latent models. Nevertheless, there remains an open issue as how to effectively exploit description documents of both users and items in matrix factorization. In this paper, we proposed dual-regularized matrix factorization with deep neural networks (DRMF) to deal with this issue. DRMF adopts a multilayered neural network model by stacking convolutional neural network and gated recurrent neural network, to generate independent distributed representations of contents of users and items. Then, representations serve to regularize the generation of latent models both for users and items in matrix factorization. We propose the corresponding algorithm for learning all parameters in DRMF. Experimental results proved that the dual-way regularization strategy significantly improves the matrix factorization methods on the accuracy of rating prediction and the recall of top-n recommendations. Also, as the components of DRMF, the new neural network model works better than the single convolutional neural network model.

Dual-regularized matrix factorization with deep neural networks for recommender systems

Traditional methods of human activity recognition usually require a large amount of strictly labeled data for training classifiers. However, it is hard for one to keep a fixed activity when collecting desired activity data by wearable sensors, and the weakly labeled data inevitably occurs in the process of data collection. For now, human activity recognition methods have seldom been researched according to weakly labeled data, which deserves deep investigation. In this paper, we proposed a novel attention-based human activity recognition method to process the weakly labeled activity data. The traditional convolutional neural network (CNN)-based human activity recognition is modified by attention mechanism, which computes the compatibility between the global features extracted at the final fully connected layers and the local features extracted at a given convolutional layer. The attention-based CNN architecture can amplify the salient activity information and suppress the irrelevant and potentially confusing information by weighing up their compatibility. Our methods are compared with two state-of-the-art methods, CNN and DeepConvLSTM. The experimental results show that our model is comparably well on the traditional UCI HAR dataset and outperforms them on the weakly labeled dataset in accuracy. Our method can greatly facilitate the process of sensor data annotation and makes data collection easier.

Attention-Based Convolutional Neural Network for Weakly Labeled Human Activities’ Recognition With Wearable Sensors

Recently, deep learning, which are able to extract automatically features from data, has achieved state-of-the-art performance across a variety of sensor based human activity recognition (HAR) tasks. However, the existing deep neural networks are usually trained with a global loss, and all hidden layer weights have to be always kept in memory before the forward and backward pass has completed. The backward locking phenomenon prevents the reuse of memory, which is a crucial limitation for wearable activity recognition. In the paper, we proposed a layer-wise convolutional neural networks (CNN) with local loss for the use of HAR task. To our knowledge, this paper is the first that uses local loss based CNN for HAR in ubiquitous and wearable computing arena. We performed experiments on five public HAR datasets including UCI HAR dataset, OPPOTUNITY dataset, UniMib-SHAR dataset, PAMAP dataset, and WISDM dataset. The results show that local loss works better than global loss for tested baseline architectures. At no extra cost, the local loss can approach the state-of-the-arts on a variety of HAR datasets, even though the number of parameters was smaller. We believe that the layer-wise CNN with local loss can be used to update the existing deep HAR methods.

Jun He

Papers

Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video

Online Robust Subspace Tracking from Partial Information

Dual-regularized matrix factorization with deep neural networks for recommender systems

Attention-Based Convolutional Neural Network for Weakly Labeled Human Activities’ Recognition With Wearable Sensors

The Layer-Wise Training Convolutional Neural Networks Using Local Loss for Sensor-Based Human Activity Recognition