Recent developments in image classification and natural language processing, coupled with the rapid growth in social media usage, have enabled fundamental advances in detecting breaking events around the world in real-time. Emergency response is one such area that stands to gain from these advances. By processing billions of texts and images a minute, events can be automatically detected to enable emergency response workers to better assess rapidly evolving situations and deploy resources accordingly. To date, most event detection techniques in this area have focused on image-only or text-only approaches, limiting detection performance and impacting the quality of information delivered to crisis response teams. In this paper, we present a new multimodal fusion method that leverages both images and texts as input. In particular, we introduce a cross-attention module that can filter uninformative and misleading components from weak modalities on a sample by sample basis. In addition, we employ a multimodal graph-based approach to stochastically transition between embeddings of different multimodal pairs during training to better regularize the learning process as well as dealing with limited training data by constructing new matched pairs from different samples. We show that our method outperforms the unimodal approaches and strong multimodal baselines by a large margin on three crisis-related tasks.

/pdf/multimodal-categorization-of-crisis-events-in-social-media-4gtsqd6gvl.pdf

Multimodal Categorization of Crisis Events in Social Media

Natural language processing for smart construction: Current status and future directions

COVID-19 outbreak: An ensemble pre-trained deep learning model for detecting informative tweets

The paper presents an overview of current and expected prospects for the development of artificial intelligence algorithms, especially in military applications, and conducted research regarding applications in the area of civilian life Attention was paid mainly to the use of AI algorithms in cybersecurity, object detection, military logistics and robotics It discusses the problems connected with the present solutions and how artificial intelligence can help solve them It briefly presents also mathematical structures and descriptions for ART, CNN and SVM networks as well as Expectation–Maximization and Gaussian Mixture Model algorithms that are used in solving of discussed problems The third chapter discusses the attitude of society towards the use of neural network algorithms in military applications The basic problems related to ethics in the application of artificial intelligence and issues of responsibility for errors made by autonomous systems are discussed

Artificial Intelligence Applications in Military Systems and Their Influence on Sense of Security of Citizens

In times of a disaster, the information available on social media can be useful for several humanitarian tasks as disseminating messages on social media is quick and easily accessible. Disaster damage assessment is inherently multi-modal, yet most existing work on damage identification has focused solely on building generic classification models that rely exclusively on text or image analysis of online social media sessions (e.g., posts). Despite their empirical success, these efforts ignore the multi-modal information manifested in social media data. Conventionally, when information from various modalities is presented together, it often exhibits complementary insights about the application domain and facilitates better learning performance. In this work, we present Crisis-DIAS, a multi-modal sequential damage identification, and severity detection system. We aim to support disaster management and aid in planning by analyzing and exploiting the impact of linguistic cues on a unimodal visual system. Through extensive qualitative, quantitative and theoretical analysis on a real-world multi-modal social media dataset, we show that the Crisis-DIAS framework is superior to the state-of-the-art damage assessment models in terms of bias, responsiveness, computational efficiency, and assessment performance.

/pdf/crisis-dias-towards-multimodal-damage-analysis-deployment-293iawga7t.pdf

Crisis-DIAS: Towards Multimodal Damage Analysis - Deployment, Challenges and Assessment

The post on information tweets increased as increase of data posted on social media during a disaster. Informative tweets can give the useful information about affected people, infrastructure damage, humanitarian organizations, etc. This paper proposed a method for classifying the informative and non-informative tweets during a disaster. The proposed approach is based on the Convolutional Neural Network (CNN) and Artificial Neural Network (ANN). CNN is used for feature extraction and ANN used as a classifier for classifying the tweets. The proposed method is tested on a real-time twitter dataset such as Hurricane Harvey 2017. Proposed method outperforms the existing methods regarding precision, recall, Fl-score and accuracy.

Detecting Informative Tweets during Disaster using Deep Neural Networks

Social media platform like Twitter is one of the primary sources for sharing real-time information at the time of events such as disasters, political events, etc. Detecting the resource tweets during a disaster is an essential task because tweets contain different types of information such as infrastructure damage, resources, opinions and sympathies of disaster events, etc. Tweets are posted related to Need and Availability of Resources (NAR) by humanitarian organizations and victims. Hence, reliable methodologies are required for detecting the NAR tweets during a disaster. The existing works don’t focus well on NAR tweets detection and also had poor performance. Hence, this paper focus on detection of NAR tweets during a disaster. Existing works often use features and appropriate machine learning algorithms on several Natural Language Processing (NLP) tasks. Recently, there is a wide use of Convolutional Neural Networks (CNN) in text classification problems. However, it requires a large amount of manual labeled data. There is no such large labeled data is available for NAR tweets during a disaster. To overcome this problem, stacking of Convolutional Neural Networks with traditional feature based classifiers is proposed for detecting the NAR tweets. In our approach, we propose several informative features such as aid, need, food, packets, earthquake, etc. are used in the classifier and CNN. The learned features (output of CNN and classifier with informative features) are utilized in another classifier (meta-classifier) for detection of NAR tweets. The classifiers such as SVM, KNN, Decision tree, and Naive Bayes are used in the proposed model. From the experiments, we found that the usage of KNN (base classifier) and SVM (meta classifier) with the combination of CNN in the proposed model outperform the other algorithms. This paper uses 2015 and 2016 Nepal and Italy earthquake datasets for experimentation. The experimental results proved that the proposed model achieves the best accuracy compared to baseline methods.

/pdf/a-stacked-convolutional-neural-network-for-detecting-the-1q34vmgwjt.pdf

A stacked convolutional neural network for detecting the resource tweets during a disaster

During the crisis, people post a large number of informative and non-informative tweets on Twitter. Informative tweets provide helpful information such as affected individuals, infrastructure damage, availability and resource requirements, etc. In contrast, non-informative tweets do not provide helpful information related to either humanitarian organizations or victims. Identifying informative tweets is a challenging task during the disaster. People often post images along with text on Twitter during the disaster. In addition to tweet text features, image features are also crucial for identifying informative tweets. However, existing methods use only text features but do not use image features to identify crisis-related tweets during the disaster. This paper proposes a novel approach by considering the image features along with the text features. It includes a text-based classification model, an image-based classification model and a late fusion. The text-based classification model uses the Convolutional Neural Network (CNN) and the Artificial Neural Network (ANN). CNN is used to extract text features from a tweet and the ANN is used to classify tweets based on extracted text features of CNN. The image-based classification model uses the fine-tuned VGG-16 architecture to extract the image features from the image and classify the image in a tweet. The output of the text-based classification model and the image-based classification model are combined using the late fusion technique to predict the tweet label. Extensive experiments are carried out on Twitter datasets of various crises, such as the Mexico earthquake, California Wildfires, etc., to demonstrate the effectiveness of the proposed method. The proposed method outperforms the state-of-the-art methods on various parameters to identify informative tweets during the disaster.

Classifying informative and non-informative tweets from the twitter by adapting image features during disaster

Nowadays, Twitter has become more popular among the users for communicating the information, especially during disaster. Identifying tweets related to the target event during disaster is a challenging task. Many prior studies discussed situational and non-situational information related to disaster. The detection of tweets related to damage assessment is a very difficult task in social media because it is a subset of situational information. One of the following drawbacks has been present in the existing damage assessment works: (1) focused only on infrastructure damage but does not include human damage in the assessment, (2) focused only on social media image data for damage assessment and (3) focused only on regional language tweets. To overcome these issues, Stacking-based Ensemble using Statistical features and Informative Words (SESIW) is proposed for detecting the tweets related to damage assessment. It uses proposed features, namely frequency of hashtags, user mentions, wh-words, URLs, count of numerals and informative words. Informative words are mined using term frequency and inverse document frequency technique. The SESIW method is tested on different Twitter disaster datasets, and it outperforms the baseline SVM with Bag-of-Words model.

Disaster damage assessment from the tweets using the combination of statistical features and informative words

In recent days, humanitarian organizations rely on social media like Twitter for situational awareness during the disaster. Millions of tweets are posted on Twitter in the form of text or images or both. Existing works showed that both image and text give complementary information during the disaster. Multi-modal informative tweet detection is helpful to both government and non-government organizations which remains a challenging task during the disaster. However, most of the existing works focused on either text or image content, but not both. In this paper, we propose a novel method based on the combination of fine-tuned BERT and DenseNet models for identifying the multi-modal informative tweets during the disaster. Fine-tuned BERT model is used to extract the linguistic, syntactic and semantic features which help deep understanding of the informative text present in the multi-modal tweet. On the other hand, the fine-tuned DenseNet model is used to extract the sophisticated features from the image. Different experiments are performed on a vast number of data-sets such as Hurricane Harvey, Hurricane Irma, Hurricane Maria, California Wildfire, Sri Lanka floods and Iraq–Iran Earthquake. Experimental results demonstrate that the proposed method outperforms the state-of-the-art method on different parameters. It is the first attempt, to the best of our knowledge, to detect multi-modal informative tweets using the combination of fine-tuned BERT and DenseNet models, where at least any text or image is informative during the disaster.

Sreenivasulu Madichetty

Papers

Detecting Informative Tweets during Disaster using Deep Neural Networks

A stacked convolutional neural network for detecting the resource tweets during a disaster

Classifying informative and non-informative tweets from the twitter by adapting image features during disaster

Disaster damage assessment from the tweets using the combination of statistical features and informative words

Multi-modal classification of Twitter data during disasters for humanitarian response