5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

AMMU: A survey of transformer-based biomedical pretrained language models

A Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis

This paper presents the development and evaluation of short-term traffic prediction models using unidirectional and bidirectional deep learning long short-term memory (LSTM) neural networks. The unidirectional LSTM (Uni-LSTM) model provides high performance through its ability to recognize longer sequences of traffic time series data. In this work, Uni-LSTM is extended to bidirectional LSTM (BiLSTM) networks which train the input data twice through forward and backward directions. The paper presents a comparative evaluation of the two models for short-term speed and traffic flow prediction using a common dataset of field observations collected from multiple freeways in Australia. The results showed BiLSTM performed better for variable prediction horizons for both speed and flow. Stacked and mixed Uni-LSTM and BiLSTM models were also investigated for 15-minute prediction horizons resulting in improved accuracy when using 4-layer BiLSTM networks. The optimized 4-layer BiLSTM model was then calibrated and validated for multiple prediction horizons using data from three different freeways. The validation results showed a high degree of prediction accuracy exceeding 90% for speeds up to 60-minute prediction horizons. For flow, the model achieved accuracies above 90% for 5- and 10-minute prediction horizons and more than 80% accuracy for 15- and 30-minute prediction horizons. These findings extend the set of AI models available for road operators and provide them with confidence in applying robust models that have been tested and evaluated on different freeways in Australia.

/pdf/unidirectional-and-bidirectional-lstm-models-for-short-term-65yb1vtmlk.pdf

Unidirectional and Bidirectional LSTM Models for Short-Term Traffic Prediction

The growth of the Internet has expanded the amount of data expressed by users across multiple platforms. The availability of these different worldviews and individuals’ emotions empowers sentiment analysis. However, sentiment analysis becomes even more challenging due to a scarcity of standardized labeled data in the Bangla NLP domain. The majority of the existing Bangla research has relied on models of deep learning that significantly focus on context-independent word embeddings, such as Word2Vec, GloVe, and fastText, in which each word has a fixed representation irrespective of its context. Meanwhile, context-based pre-trained language models such as BERT have recently revolutionized the state of natural language processing. In this work, we utilized BERT’s transfer learning ability to a deep integrated model CNN-BiLSTM for enhanced performance of decision-making in sentiment analysis. In addition, we also introduced the ability of transfer learning to classical machine learning algorithms for the performance comparison of CNN-BiLSTM. Additionally, we explore various word embedding techniques, such as Word2Vec, GloVe, and fastText, and compare their performance to the BERT transfer learning strategy. As a result, we have shown a state-of-the-art binary classification performance for Bangla sentiment analysis that significantly outperforms all embedding and algorithms.

/pdf/transfer-learning-for-sentiment-analysis-using-bert-based-2kkw6etv.pdf

Transfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning

The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones as well as composed ones and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification and Named Entity Recognition tasks.

ParsBERT: Transformer-based Model for Persian Language Understanding

The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones and gathered ones, and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification, and Named Entity Recognition tasks.

Traffic flow characteristics are one of the most critical decision-making and traffic policing factors in a region. Awareness of the predicted status of the traffic flow has prime importance in traffic management and traffic information divisions. The purpose of this research is to suggest a forecasting model for traffic flow by using deep learning techniques based on historical data in the Intelligent Transportation Systems area. The historical data collected from the Caltrans Performance Measurement Systems (PeMS) for six months in 2019. The proposed prediction model is a Variational Long Short-Term Memory Encoder in brief VLSTM-E try to estimate the flow accurately in contrast to other conventional methods. VLSTM-E can provide more reliable short-term traffic flow by considering the distribution and missing values.

Marzieh Farahani

Papers

ParsBERT: Transformer-based Model for Persian Language Understanding

ParsBERT: Transformer-based Model for Persian Language Understanding

Short-Term Traffic Flow Prediction Using Variational LSTM Networks