https://pure.mpg.de/pubman/item/item_1819068_3/component/file_1840695/MPI-I-2007-5-003.pdf

YAGO: A Large Ontology from Wikipedia and WordNet

The field of machine learning has matured to the point where many sophisticated learning approaches can be applied to practical applications. Thus it is of critical importance that researchers have the proper tools to evaluate learning approaches and understand the underlying issues. This book examines various aspects of the evaluation process with an emphasis on classification algorithms. The authors describe several techniques for classifier performance assessment, error estimation and resampling, obtaining statistical significance as well as selecting appropriate domains for evaluation. They also present a unified evaluation framework and highlight how different components of evaluation are both significantly interrelated and interdependent. The techniques presented in the book are illustrated using R and WEKA facilitating better practical insight as well as implementation.Aimed at researchers in the theory and applications of machine learning, this book offers a solid basis for conducting performance evaluations of algorithms in practical settings.

Evaluating Learning Algorithms: A Classification Perspective

The large number of potential applications from bridging web data with knowledge bases have led to an increase in the entity linking research. Entity linking is the task to link entity mentions in text with their corresponding entities in a knowledge base. Potential applications include information extraction, information retrieval, and knowledge base population. However, this task is challenging due to name variations and entity ambiguity. In this survey, we present a thorough overview and analysis of the main approaches to entity linking, and discuss various applications, the evaluation of entity linking systems, and future directions.

/pdf/entity-linking-with-a-knowledge-base-issues-techniques-and-2ds9mn4o8g.pdf

Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions

The Nature Of Emotion Fundamental Questions

Background: Since the beginning of December 2019, the coronavirus disease (COVID-19) has spread rapidly around the world, which has led to increased discussions across online platforms. These conversations have also included various conspiracies shared by social media users. Amongst them, a popular theory has linked 5G to the spread of COVID-19, leading to misinformation and the burning of 5G towers in the United Kingdom. The understanding of the drivers of fake news and quick policies oriented to isolate and rebate misinformation are keys to combating it.
Objective: The aim of this study is to develop an understanding of the drivers of the 5G COVID-19 conspiracy theory and strategies to deal with such misinformation.
Methods: This paper performs a social network analysis and content analysis of Twitter data from a 7-day period (Friday, March 27, 2020, to Saturday, April 4, 2020) in which the #5GCoronavirus hashtag was trending on Twitter in the United Kingdom. Influential users were analyzed through social network graph clusters. The size of the nodes were ranked by their betweenness centrality score, and the graph’s vertices were grouped by cluster using the Clauset-Newman-Moore algorithm. The topics and web sources used were also examined.
Results: Social network analysis identified that the two largest network structures consisted of an isolates group and a broadcast group. The analysis also revealed that there was a lack of an authority figure who was actively combating such misinformation. Content analysis revealed that, of 233 sample tweets, 34.8% (n=81) contained views that 5G and COVID-19 were linked, 32.2% (n=75) denounced the conspiracy theory, and 33.0% (n=77) were general tweets not expressing any personal views or opinions. Thus, 65.2% (n=152) of tweets derived from nonconspiracy theory supporters, which suggests that, although the topic attracted high volume, only a handful of users genuinely believed the conspiracy. 

This paper also shows that fake news websites were the most popular web source shared by users; although, YouTube videos were also shared. The study also identified an account whose sole aim was to spread the conspiracy theory on Twitter.
Conclusions: The combination of quick and targeted interventions oriented to delegitimize the sources of fake information is key to reducing their impact. Those users voicing their views against the conspiracy theory, link baiting, or sharing humorous tweets inadvertently raised the profile of the topic, suggesting that policymakers should insist in the efforts of isolating opinions that are based on fake news. Many social media platforms provide users with the ability to report inappropriate content, which should be used. This study is the first to analyze the 5G conspiracy theory in the context of COVID-19 on Twitter offering practical guidance to health authorities in how, in the context of a pandemic, rumors may be combated in the future.

/pdf/covid-19-and-the-5g-conspiracy-theory-social-network-3589sws8pz.pdf

COVID-19 and the 5G Conspiracy Theory: Social Network Analysis of Twitter Data.

News content can sometimes be misleading and influence users’ decision making processes (e.g., voting decisions). Quantitatively assessing the truthfulness of content becomes key, but it is often challenging and thus done by experts. In this work we look at how experts and non-expert assess truthfulness of content by focusing on the effect of the adopted judgment scale and of assessors’ own bias on the judgments they perform. Our results indicate a clear effect of the assessors’ political background on their judgments where they tend to trust content which is aligned to their own belief, even if experts have marked it as false. Crowd assessors also seem to have a preference towards coarse-grained scales, as they tend to use a few extreme values rather than the full breadth of fine-grained scales.

/pdf/crowdsourcing-truthfulness-the-impact-of-judgment-scale-and-1wea0gfg1t.pdf

Crowdsourcing truthfulness: the impact of judgment scale and assessor bias

Many systems require access to very large amounts of data to properly function, like systems allowing to visualize or predict meteorological changes in a country over a given period of time, or any other system holding, processing and displaying scientific or sensor data. However, filling out a database with large amounts of valuable data can be a difficult, costly and time-consuming task. In this paper, we present techniques to create large amounts of data by combining crowdsourcing, data generation models, mobile computing, and big data analytics. We have implemented our methods in a system, NoizCrowd, allowing to crowdsource noise levels in a given region and to generate noise models by using state-of-the-art noise propagation models and array data management techniques. The resulting models and data can then be accessed using a visual interface.

/pdf/noizcrowd-a-crowd-based-data-gathering-and-management-system-up2imf3ow5.pdf

NoizCrowd: A Crowd-Based Data Gathering and Management System for Noise Level Data

The dominant approach to evaluate the effectiveness of information retrieval (IR) systems is by means of reusable test collections built following the Cranfield paradigm. In this paper, we propose a new IR evaluation methodology based on pooled test-collections and on the continuous use of either crowdsourcing or professional editors to obtain relevance judgements. Instead of building a static collection for a finite set of systems known a priori, we propose an IR evaluation paradigm where retrieval approaches are evaluated iteratively on the same collection. Each new retrieval technique takes care of obtaining its missing relevance judgements and hence contributes to augmenting the overall set of relevance judgements of the collection. We also propose two metrics: Fairness Score, and opportunistic number of relevant documents, which we then use to define new pooling strategies. The goal of this work is to study the behavior of standard IR metrics, IR system ranking, and of several pooling techniques in a continuous evaluation context by comparing continuous and non-continuous evaluation results on classic test collections. We both use standard and crowdsourced relevance judgements, and we actually run a continuous evaluation campaign over several existing IR systems.

/pdf/pooling-based-continuous-evaluation-of-information-retrieval-22bmbuhhn8.pdf

Pooling-based continuous evaluation of information retrieval systems

Named Entity Recognition (NER) plays an important role in a variety of online information management tasks including text categorization, document clustering, and faceted search. While recent NER systems can achieve near-human performance on certain documents like news articles, they still remain highly domain-specific and thus cannot effectively identify entities such as original technical concepts in scientific documents. In this work, we propose novel approaches for NER on distinctive document collections (such as scientific articles) based on n-grams inspection and classification. We design and evaluate several entity recognition features---ranging from well-known part-of-speech tags to n-gram co-location statistics and decision trees---to classify candidates. In addition, we show how the use of external knowledge bases (either specific like DBLP or generic like DBPedia) can be leveraged to improve the effectiveness of NER for idiosyncratic collections. We evaluate our system on two test collections created from a set of Computer Science and Physics papers and compare it against state-of-the-art supervised methods. Experimental results show that a careful combination of the features we propose yield up to 85% NER accuracy over scientific collections and substantially outperforms state-of-the-art approaches such as those based on maximum entropy.

Effective named entity recognition for idiosyncratic web collections

With the growth of the Web and the variety of search engine users, Web search efectiveness and user satisfaction can be improved by diversirication. This paper surveys recent approaches to search result diversification in both full text and structured content search. We identify commonalities in the proposed methods describing an overall framework for result diversification. We discuss diferent diversity dimensions and measures as well as possible ways of considering the relevance / diversity trade-off. We also summarise existing efforts evaluating diversity in search. Moreover, for each of these steps, we point out aspects which are missing in current approaches as possible directions for future work.

/pdf/current-approaches-to-search-result-diversification-2n6bj0fa1n.pdf

Gianluca Demartini

Papers

Crowdsourcing truthfulness: the impact of judgment scale and assessor bias

NoizCrowd: A Crowd-Based Data Gathering and Management System for Noise Level Data

Pooling-based continuous evaluation of information retrieval systems

Effective named entity recognition for idiosyncratic web collections

Current approaches to search result diversification