A noisy label and negative sample robust loss function for DNN-based distant supervised relation extraction

Sporting events generate a massive amount of traffic on social media with live moment-to-moment accounts as any given situation unfolds. The generated data are intensified by fans feelings, reactions, and subjective opinions towards what happens during the event, all of which are based on their individual points of view. Analyzing and summarizing this data will generate a comprehensive overview of the event in terms of how the event evolves and how fans react and view the event based on their perspectives. Previously, most of the summarization works ignore fan reactions and subjective opinions, and focus primarily on generating an objective-view summary. We believe that an effective and useful summary should consider human reactions, sentiment, and point of view, as opposed to simply describing what happens during the event. Accordingly, in this work, we propose MMSUM Digital Twins: a summarization framework that is capable of generating a multi-view multi-modal summary for sporting events in real-time. The proposed digital twins-based framework consists of four main components: sub-event recognition which detects the event’s key moments, tweet categorization, which determines which team the tweets’ writers support and assigns tweets to their teams, sentiment analysis to track fans’ state of mind, and image popularity prediction for selecting representative images. Furthermore, the MMSUM employs a visual-filtering model to address the issue of noisy images that inundate social media, compromising the summarization quality. We leverage the knowledge of sport fans to evaluate the generated multi-view summarization through an online user study. The experiment results confirm the effectiveness of our proposed approach for summarizing sporting events by considering multimedia data, sentiment, and subjective views of the event.

MMSUM Digital Twins: A Multi-view Multi-modality Summarization Framework for Sporting Events

Real-world events generate a massive amount of traffic on social media with live, moment-to-moment accounts as any given situation unfolds. The data generated on social media is loaded with the sentiment, opinions, and reactions of the public towards the events. Browsing the event-related data in its raw form would be an overwhelming task due to the extensive amount of data, making the search for any specific updates or useful information an especially daunting and time-consuming endeavor. Event-related data, if effectively summarized, could generate a comprehensive overview of the event in terms of what happened and how people reacted at the time. Unfortunately, most of the eventbased summarization systems concentrate their effort on simply describing what happened during the event; ignoring a valuable resource of emotional reactions that portrays the event from varying perspectives. Accordingly, in this thesis, we introduce an event-based summarization approach that incorporates multimedia data, sentiment, and human reactions with respect to their point of view in order to generate a concise subjective multimedia summary of the event. In order to achieve our goal, we introduce popularity prediction and sentiment analysis models, both of which are essential in our summarization approach. As the event unfolds over time, we utilize the popularity prediction model to extract a representative set of images for the event to be included in the summary based on the predicted popularity scores. Multi-modality features are exploited to develop our model, including various levels of visual features, textual features, and contextual features. In order to track users’ opinions and changes in their sentiment in correlation with the occurrence of sub-events, we develop a sentiment classification model that effectively recognizes the sentiment conveyed in sport conversations. Due to the lack of a manually annotated sentiment dataset, we propose a new, manually labeled football-based sentiment dataset. We also create an automatically generated sentiment-lexicon specifically for the football domain. We assess the performance of our developed models and generated summary through extensive experimental evaluation. We explore the impact of different features on the performance of the popularity prediction and sentiment models. We also leverage the knowledge of sport fans to evaluate the generated subjective summarization through an online user-based survey. The experiment results confirm the effectiveness of our proposed solution for event-based summarization by considering multimedia data, sentiment, and people’s subjective views of events.

/pdf/towards-subjective-multimedia-summarization-framework-for-4brc469wyh.pdf

Towards Subjective Multimedia Summarization Framework for Sporting Event in the Context of Digital Twins

The connection between news and the images that illustrate them goes beyond visual concept to natural language matching. Instead, the open-domain and event-reporting nature of news leads to semantically complex texts, in which images are used as a contextualizing element. This connection is often governed by a certain level of indirection, with journalistic criteria also playing an important role. In this paper, we address the complex challenge of connecting images to news text. A context-enriched Multimodal Transformer model is proposed, NewsLXMERT, capable of jointly attending to complementary multimodal news data perspectives. The idea is to create knowledge-rich and diverse multimodal sequences, going beyond the news headline (often lacking the necessary context) and visual objects, to effectively ground images to news pieces. A comprehensive evaluation of challenging image-news piece matching settings is conducted, where we show the effectiveness of NewsLXMERT, the importance of leveraging the additional context and demonstrate the usefulness of the obtained pre-trained news representations for transfer-learning. Finally, to shed light on the heterogeneous nature of the problem, we contribute with a systematic model-driven study that identifies image-news matching profiles, thus explaining news piece-image matches.

Understanding News Text and Images Connection with Context-enriched Multimodal Transformers

Creating a cohesive, high-quality, relevant, media story is a challenge that news media editors face on a daily basis. This challenge is aggravated by the flood of highly relevant information that is constantly pouring onto the newsroom. To assist news media editors in this daunting task, this paper proposes a framework to organize news content into cohesive, high-quality, relevant visual storylines. First, we formalize, in a nonsubjective manner, the concept of visual story transition. Leveraging it, we propose four graph-based methods of storyline creation, aiming for global story cohesiveness. These were created and implemented to take full advantage of existing graph algorithms, ensuring their correctness and good computational performance. They leverage a strong ensemble-based estimator which was trained to predict story transition quality based on both the semantic and visual features present in the pair of images under scrutiny. A user study covered a total of 28 curated stories about sports and cultural events. Experiments showed that (i) visual transitions in storylines can be learned with a quality above 90%, and (ii) the proposed graph methods can produce cohesive storylines with quality in the range of 88% to 96%.

Assisting News Media Editors with Cohesive Visual Storylines.

News editors need to find the photos that best illustrate a news piece and fulfill news-media quality standards, while being pressed to also find the most recent photos of live events. Recently, it became common to use social-media content in the context of news media for its unique value in terms of immediacy and quality. Consequently, the amount of images to be considered and filtered through is now too much to be handled by a person. To aid the news editor in this process, we propose a framework designed to deliver high-quality, news-press type photos to the user. The framework, composed of two parts, is based on a ranking algorithm tuned to rank professional media highly and a visual SPAM detection module designed to filter-out low-quality media. The core ranking algorithm is leveraged by aesthetic, social and deep-learning semantic features. Evaluation showed that the proposed framework is effective at finding high-quality photos (true-positive rate) achieving a retrieval MAP of 64.5% and a classification precision of 70%.

Ricardo Pinto

Papers

Ranking News-Quality Multimedia

Ranking News-Quality Multimedia