Journal ArticleDOI
Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm
A. P. Dawid,A. M. Skene +1 more
TLDR
The EM algorithm is shown to provide a slow but sure way of obtaining maximum likelihood estimates of the parameters of interest in compiling a patient record.Abstract:
In compiling a patient record many facets are subject to errors of measurement. A model is presented which allows individual error-rates to be estimated for polytomous facets even when the patient's "true" response is not available. The EM algorithm is shown to provide a slow but sure way of obtaining maximum likelihood estimates of the parameters of interest. Some preliminary experience is reported and the limitations of the method are described.read more
Citations
More filters
Proceedings ArticleDOI
Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
TL;DR: This work explores the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web, and proposes a technique for bias correction that significantly improves annotation quality on two tasks.
Posted Content
Concrete Problems in AI Safety
TL;DR: A list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function, an objective function that is too expensive to evaluate frequently, or undesirable behavior during the learning process, are presented.
Journal ArticleDOI
Classification in the Presence of Label Noise: A Survey
Benoît Frénay,Michel Verleysen +1 more
TL;DR: In this paper, label noise consists of mislabeled instances: no additional information is assumed to be available like e.g., confidences on labels.
Journal ArticleDOI
Learning From Crowds
Vikas C. Raykar,Shipeng Yu,Linda Zhao,Gerardo Hermosillo Valadez,Charles Florin,Luca Bogoni,Linda Moy +6 more
TL;DR: A probabilistic approach for supervised learning when the authors have multiple annotators providing (possibly noisy) labels but no absolute gold standard, and experimental results indicate that the proposed method is superior to the commonly used majority voting baseline.
Journal ArticleDOI
A brief introduction to weakly supervised learning
TL;DR: This article reviews some research progress of weakly supervised learning, focusing on three typical types of weak supervision: incomplete supervision, where only a subset of training data is given with labels; inexact supervision, Where the training data are given with only coarse-grained labels; and inaccurate supervision,Where the given labels are not always ground-truth.
References
More filters
Journal ArticleDOI
The measurement of observer agreement for categorical data
J. R. Landis,Gary G. Koch +1 more
TL;DR: A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Journal ArticleDOI
Latent Structure Analysis.
Journal ArticleDOI
Exploratory latent structure analysis using both identifiable and unidentifiable models
TL;DR: In this article, the authors considered a wide class of latent structure models, which can serve as possible explanations of the observed relationships among a set of m manifest polytomous variables.
Journal ArticleDOI
A review of statistical methods in the analysis of data arising from observer reliability studies (Part II)
J. Richard Landis,Gary G. Koch +1 more
TL;DR: In this article, a review of the literature in observer variability is surveyed with attention given to a notational unification of the various models proposed, and measures of agreement, such as kappa and weighted-kappa, are discussed in the context of nominal and ordinal data.