Truth Inference in Crowdsourcing: Is the Problem Solved?
Yudian Zheng
†
, Guoliang Li
#
, Yuanbing Li
#
, Caihua Shan
†
, Reynold Cheng
†
#
Department of Computer Science, Tsinghua University
†
Department of Computer Science, The University of Hong Kong
ydzheng2@cs.hku.hk, liguoliang@tsinghua.edu.cn
yb-li16@mails.tsinghua.edu.cn, chshan@cs.hku.hk, ckcheng@cs.hku.hk
ABSTRACT
Crowdsourcing has emerged as a novel problem-solving paradigm,
which facilitates addressing problems that are hard for comput-
ers, e.g., entity resolution and sentiment analysis. However, due to
the openness of crowdsourcing, workers may yield low-quality an-
swers, and a redundancy-based method is widely employed, which
first assigns each task to multiple workers and then infers the cor-
rect answer (called truth) for the task based on the answers of the
assigned workers. A fundamental problem in this method is Truth
Inference, which decides how to effectively infer the truth. Re-
cently, the database community and data mining community in-
dependently study this problem and propose various algorithms.
However, these algorithms are not compared extensively under the
same framework and it is hard for practitioners to select appropriate
algorithms. To alleviate this problem, we provide a detailed survey
on 17 existing algorithms and perform a comprehensive evaluation
using 5 real datasets. We make all codes and datasets public for
future research. Through experiments we find that existing algo-
rithms are not stable across different datasets and there is no algo-
rithm that outperforms others consistently. We believe that the truth
inference problem is not fully solved, and identify the limitations
of existing algorithms and point out promising research directions.
1. INTRODUCTION
Crowdsourcing solutions have been proposed to address tasks
that are hard for machines, e.g., entity resolution [8] and sentiment
analysis [32]. Due to the wide deployment of public crowdsourc-
ing platforms, e.g., Amazon Mechanical Turk (AMT) [2], Crowd-
Flower [12], the access to crowd becomes much easier. As reported
in [1], more than 500K workers from 190 countries have performed
tasks on AMT [2]. The database community has shown great in-
terests in crowdsourcing (see a survey [29]). Several crowdsourced
databases (e.g., CrowdDB [20], Deco [39], Qurk [37]) are built to
incorporate the crowd into query processing, and there are many
studies on implementing crowdsourced operators, e.g., Join [50,
36, 52, 11], Max [47, 22], Top-k [14, 55], Group-by [14], etc.
Due to the openness of crowdsourcing, the crowd (called work-
ers) may yield low-quality or even noisy answers. Thus it is impor-
This work is licensed under the Creative Commons Attribution-
NonCommercial-NoDerivatives 4.0 International License. To view a copy
of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. For
any use beyond those covered by this license, obtain permission by emailing
info@vldb.org.
Proceedings of the VLDB Endowment, Vol. 10, No. 5
Copyright 2017 VLDB Endowment 2150-8097/17/01.
tant to control the quality in crowdsourcing. To address this prob-
lem, most of existing crowdsourcing studies employ a redundancy-
based strategy, which assigns each task to multiple workers and
aggregates the answers given by different workers to infer the cor-
rect answer (called truth) of each task. A fundamental problem,
called Truth Inference, is widely studied in existing crowdsourcing
works [34, 16, 15, 53, 51, 41, 26, 33, 61, 19, 35, 30, 27, 10, 46, 5,
31], which decides how to effectively infer the truth for each task.
To address the problem, a straightforward approach is Majority
Voting (MV), which takes the answer given by majority workers
as the truth. However, the biggest limitation of MV is that it re-
gards all workers as equal. In reality, workers may have different
levels of qualities: a high-quality worker carefully answers tasks;
a low-quality (or spammer) may randomly answer tasks in order
to deceive money; a malicious worker may even intentionally give
wrong answers. Thus it is important to capture each worker’s qual-
ity, which can better infer the truth of each task by trusting more on
the answers given by workers with higher qualities.
However, the ground truth of each task is unknown and it is hard
to estimate a worker’s quality. To address this problem, one can
label the ground truth for a small portion of tasks (called golden
tasks) and use them to estimate workers’ quality. There are two
types of methods to utilize golden tasks. The first is qualification
test. Each worker requires to perform a set of golden tasks before
she can really answer tasks, and her quality is computed based on
her answering performance for these golden tasks. The second is
hidden test. The golden tasks are mixed into the tasks and the
workers do not know which are golden tasks. A worker’s quality
is computed based on her answering performance on these golden
tasks. However, the two approaches have some limitations. (1)
For qualification test, workers require to answer these “extra” tasks
without pay, and many workers do not want to answer such tasks.
(2) For hidden test, it is a waste to pay the “extra” tasks. (3) The
two techniques may not improve the quality (see Section 6).
Considering these limitations, the database community [34, 19,
35, 24, 30, 31, 58] and data mining community [16, 53, 15, 61,
27, 46, 41, 51, 26, 33, 5] independently study this problem and
propose various algorithms. However, these algorithms are not
compared under the same experimental framework and it is hard
for practitioners to select appropriate algorithms. To alleviate this
problem, we provide a comprehensive survey on existing truth in-
ference algorithms. We summarize them in terms of task types, task
modeling, worker modeling, and inference techniques. We conduct
a comprehensive comparison of 17 existing representative meth-
ods [16, 53, 15, 61, 27, 46, 41, 30, 5, 31, 51, 26, 33], experimen-
tally compare them on 5 real datasets with varying sizes and task
types in real crowdsourcing platforms, make a deep analysis on the
experimental results, and provide extensive experimental findings.
541
Table 1: A Product Dataset.
ID Product Name
r
1
iPad Two 16GB WiFi White
r
2
iPad 2nd generation 16GB WiFi White
r
3
Apple iPhone 4 16GB White
r
4
iPhone 4th generation White 16GB
Table 2: Collected Workers’ Answers for All Tasks.
t
1
: t
2
: t
3
: t
4
: t
5
: t
6
:
(r
1
=r
2
) (r
1
=r
3
) (r
1
=r
4
) (r
2
=r
3
) (r
2
=r
4
) (r
3
=r
4
)
w
1
F T T F F F
w
2
F F T T F
w
3
T F F F F T
To summarize, we make the following contributions:
• We survey 17 existing algorithms, summarize a framework (Sec-
tion 3), and provide an in-depth analysis and summary on the 17
algorithms in different perspectives (Sections 4-5), which can help
practitioners to easily grasp existing truth inference algorithms.
• We experimentally conduct a thorough comparison of these meth-
ods on 5 datasets with varying sizes, publicize our codes and datasets
[40], and provide experimental findings, which give guidance for
selecting appropriate methods under various scenarios (Section 6).
• We find that the truth inference problem is not fully solved, iden-
tify the limitations of existing algorithms, and point out several
promising research directions (Section 7).
2. PROBLEM DEFINITION
DEFINITION 1 (TASK). A task set T contains n tasks, i.e.,
T = {t
1
, t
2
, . . . , t
n
}. Each task asks workers to answer the task.
Existing studies mainly focus on three types of tasks.
Decision-Making Tasks. A decision-making task has a claim and
asks workers to make a decision on whether the claim is true (de-
noted as ‘T’) or false (denoted as ‘F’). Decision-making tasks are
widely used and studied in existing crowdsourcing works [34, 16,
15, 53, 51, 41, 26, 33, 61, 19, 35, 30, 27, 46, 5] because of its
conceptual simplicity.
Next we take entity resolution as an example, which tries to find
pairs of products in Table 1 that refer to the same real-world en-
tity. A straightforward way is to generate a task set T = {(r
1
=r
2
),
(r
1
=r
3
), (r
1
=r
4
), (r
2
=r
3
), (r
2
=r
4
), (r
3
=r
4
)} with n = 6 decision-
making tasks, where each task has two choices: (true, false), and
asks workers to select a choice for the task. For example, t
2
(or
r
1
=r
3
) asks whether the claim ‘iPad Two 16GB WiFi White = Ap-
ple iPhone 4 16GB White’ is true (‘T’) or false (‘F’). Tasks are then
published to crowdsourcing platforms (e.g., AMT [2]) and work-
ers’ answers are collected.
Single-Choice (and Multiple-Choice) Tasks. A single-choice task
contains a question and a set of candidate choices, and asks work-
ers to select a single choice out of the candidate choices. For ex-
ample, in sentiment analysis, a task asks workers to select the sen-
timent (‘positive’, ‘neutral’, ‘negative’) of a given tweet. Decision-
making task is a special case of single-choice task, with two special
choices (‘T’ and ‘F’). The single-choice tasks are especially studied
in [34, 16, 15, 53, 41, 61, 35, 30, 27, 46, 5]. A direct extension of
single-choice task is multiple-choice task, where workers can select
multiple choices (not only a single choice) out of a set of candidate
choices. For example, in image tagging, given a set of candidate
tags for an image, it asks workers to select the tags that the im-
age contains. However, as addressed in [60, 38], a multiple-choice
task can be easily transformed to a set of decision-making tasks,
e.g., for an image tagging task (multiple-choice), each transformed
decision-making task asks whether or not a tag is contained in an
image. Thus the methods in decision-making tasks can be directly
extended to handle multiple-choice tasks.
Table 3: Notations.
Notation Description
t
i
the i-th task (1 ≤ i ≤ n) and T = {t
1
, t
2
, . . . , t
n
}
w the worker w and W = {w} is the set of workers
W
i
the set of workers that have answered task t
i
T
w
the set of tasks that have been answered by worker w
v
w
i
the answer given by worker w for task t
i
V the set of workers’ answers for all tasks, i.e., V = {v
w
i
}
v
∗
i
the (ground) truth for task t
i
(1 ≤ i ≤ n)
Numeric Tasks. The numeric task asks workers to provide a value.
For example, a task asks about the height of Mount Everest. Dif-
ferent from the tasks above, workers’ inputs are numeric values,
which have inherent orderings (e.g., compared with 8800m, 8845m
is closer to 8848m). Existing works [41, 30] especially study such
tasks by considering the inherent orderings between values.
Others. Besides the above tasks, there are other types of tasks,
e.g., translate a language to another [10], or ask workers to collect
data (e.g., the name of a celebrity) [20, 48]. However, it is hard
to control the quality for such “open” tasks. Thus they are rarely
studied in existing works [10, 20, 48]. In this paper, we focus only
on the above three tasks and leave other tasks for future work.
DEFINITION 2 (WORKER). A worker set W contains a set of
workers, i.e., W = {w}. Let W
i
denote the set of workers that
have answered task t
i
and T
w
denote the set of tasks that have
been answered by worker w.
DEFINITION 3 (ANSWER). Each task t
i
can be answered with
a subset of workers in W. Let v
w
i
denote the worker w’s answer
for task t
i
, and the set of answers V = {v
w
i
} contains the collected
workers’ answers for all tasks.
Table 2 shows an example, with answers to T given by three
workers W = {w
1
, w
2
, w
3
}. (The empty cell means that the
worker does not answer the task.) For example, v
w
1
4
= F means
worker w
1
answers t
4
(i.e., r
2
= r
3
) with ‘F’, i.e., w
1
thinks that
r
2
6= r
3
. The set of workers that answer t
1
is W
1
= {w
1
, w
3
}, and
the set of tasks answered by worker w
2
is T
w
2
= {t
2
, t
3
, t
4
, t
5
, t
6
}.
DEFINITION 4 (TRUTH). Each task t
i
has a true answer, called
the ground truth (or truth), denoted as v
∗
i
.
For the example task set T in Table 1, only pairs (r
1
= r
2
) and
(r
3
= r
4
) are true, and thus v
∗
1
= v
∗
6
= T, and others’ truth are F.
Based on the above notations, the truth inference problem is to
infer the (unknown) truth v
∗
i
for each task t
i
based on V .
DEFINITION 5 (TRUTH INFERENCE IN CROWDSOURCING).
Given workers’ answers V , infer the truth v
∗
i
of each task t
i
∈ T .
Table 3 summarizes the notations used in the paper.
3. SOLUTION FRAMEWORK
A naive solution is Majority Voting (MV) [20, 39, 37], which re-
gards the choice answered by majority workers as the truth. Based
on Table 2, the truth derived by MV is v
∗
i
= F for 2 ≤ i ≤ 6 and
it randomly infers v
∗
1
to break the tie. The MV incorrectly infers
v
∗
6
, and has 50% chance to infer v
∗
1
wrongly. The reason is that MV
assumes that each worker has the same quality, and in reality, work-
ers have different qualities: some are experts or ordinary workers,
while others are spammers (who randomly answer tasks in order to
deceive money) or even malicious workers (who intentionally give
wrong answers). Take a closer look at Table 2, we can observe that
w
3
has a higher quality, and the reason is that if we do not consider
t
1
(which receives 1 ‘T’ and 1 ‘F’), then w
3
gives 4 out of 5 an-
swers that are reported by majority workers, while w
1
and w
2
give
both 3 out of 5, thus we should give higher trust to w
3
’s answer and
in this way can infer all tasks’ truth correctly.
542
Based on the above discussions, existing works [16, 15, 53, 51,
41, 33, 26, 61, 62, 19, 35, 30, 46, 27, 5, 34] propose various ways
to model a worker’s quality. Although qualification test and hidden
test can help to estimate a worker’s quality, they require to label
tasks with truth beforehand, and a worker also requires to answer
these “extra” tasks. To address this problem, existing works [16,
15, 53, 51, 41, 33, 26, 61, 62, 19, 35, 30, 46, 27, 5, 34] estimate
each worker’s quality purely based on workers’ answers V . Intu-
itively, they capture the inherent relations between workers’ quali-
ties and tasks’ truth: for a task, the answer given by a high-quality
worker is highly likely to be the truth; conversely, for a worker,
if the worker often correctly answers tasks, then the worker will
be assigned with a high quality. By capturing such relations, they
adopt an iterative approach, which jointly infers both the workers’
qualities and tasks’ truth.
By capturing the above relations, the general approach adopted
by most of existing works [16, 15, 53, 51, 41, 33, 26, 61, 62, 19,
35, 30, 46, 27, 5, 34] is shown in Algorithm 1. The quality of each
worker w ∈ W is denoted as q
w
. In Algorithm 1, it first initializes
workers’ qualities randomly or using qualification test (line 1), and
then adopts an iterative approach with two steps (lines 3-11):
Step 1: Inferring the Truth (lines 3-5): it infers each task’s truth
based on workers’ answers and qualities. In this step, different task
types are handled differently. Furthermore, some existing works [53,
51] explicitly model each task, e.g., [53] regards that different tasks
may have different difficulties. We discuss how existing works
model a task in Section 4.1.
Step 2: Estimating Worker Quality (lines 6-8): based on work-
ers’ answers and each task’s truth (derived from step 1), it esti-
mates each worker’s quality. In this step, existing works model
each worker w’s quality q
w
differently. For example, [16, 26, 33,
5] model q
w
as a single value, while [15, 41, 33, 27, 46] model q
w
as a matrix. We discuss worker’s models in Section 4.2.
Convergence (lines 9-11): the two iterations will run until conver-
gence. Typically to identify convergence, existing works will check
whether the change of two sets of parameters (i.e., workers’ quali-
ties and tasks’ truth) is below some defined threshold (e.g., 10
−3
).
Finally the inferred truth and workers’ qualities are returned.
Running Example. Let us show how the method PM [31, 5] works
for Table 2. PM models each worker w as a single value q
w
∈
[0, +∞) and a higher value implies a higher quality. Initially, each
worker w ∈ W is assigned with the same quality q
w
= 1. Then
the two steps devised in PM are as follows:
Step 1 (line 5): v
∗
i
= argmax
v
P
w∈W
i
q
w
· 1
{v=v
w
i
}
;
Step 2 (line 8): q
w
= − log
P
t
i
∈T
w
1
{v
∗
i
6=v
w
i
}
max
w∈W
{
P
t
i
∈T
w
1
{v
∗
i
6=v
w
i
}
}
.
The indicator function 1
{·}
returns 1 if the statement is true; 0,
otherwise. For example, 1
{5=3}
= 0 and 1
{5=5}
= 1. For the 1st
iteration, in step 1, it computes each task’s truth from workers’ an-
swers by considering which choice receives the highest aggregated
workers’ qualities. Intuitively, the answer given by many high qual-
ity workers are likely to be the truth. For example, for task t
2
, as
it receives one T and two F’s from workers and each worker is of
the same quality, then v
∗
2
= F. Similarly we get v
∗
1
= T and v
∗
i
= F
for 2 ≤ i ≤ 6. In step 2, based on the computed truth in step
1, it gives a high (low) quality to a worker if the worker makes
few (a lot of) mistakes. For example, as the number of mistakes
(i.e.,
P
t
i
∈T
w
1
{v
∗
i
6=v
w
i
}
) for workers w
1
, w
2
, w
3
are 3, 2, 1, re-
spectively, thus the computed qualities are q
w
1
= − log(3/3) = 0,
q
w
2
= − log(2/3) = 0.41 and q
w
3
= − log(1/3) = 1.10. Follow-
ing these two steps, the process will then iterate until convergence.
In the converged results, the truth are v
∗
1
= v
∗
6
= T, and v
∗
i
= F
Algorithm 1: Solution Framework
Input: workers’ answers V
Output: inferred truth v
∗
i
(1 ≤ i ≤ n), worker quality q
w
(w ∈ W)
1 Initialize all workers’ qualities (q
w
for w ∈ W);
2 while true do
3 // Step 1: Inferring the Truth
4 for 1 ≤ i ≤ n do
5 Inferring the truth v
∗
i
based on V and {q
w
| w ∈ W};
6 // Step 2: Estimating Worker Quality
7 for w ∈ W do
8 Estimating the quality q
w
based on V and {v
∗
i
| 1 ≤ i ≤ n};
9 // Check for Convergence
10 if Converged then
11 break;
12 return v
∗
i
for 1 ≤ i ≤ n and q
w
for w ∈ W;
(2 ≤ i ≤ 5); the qualities are q
w
1
= 4.9 × 10
−15
, q
w
2
= 0.29 and
q
w
3
= 16.09. We can observe that PM can derive the truth correctly,
and w
3
has a higher quality compared with w
1
and w
2
.
4. IMPORTANT FACTORS
In this section, we categorize existing works [16, 15, 53, 51, 41,
33, 26, 61, 62, 19, 35, 30, 46, 27, 5, 34] following two factors:
Task Modeling (Section 4.1): how existing works model a task
(e.g., task’s difficulty, latent topics).
Worker Modeling (Section 4.2): how existing works model a worker’s
quality (e.g., worker probability, diverse skills).
We summarize how existing works [16, 15, 53, 51, 41, 33, 26,
61, 62, 19, 35, 30, 46, 27, 5, 34] can be categorized based on the
above factors in Table 4. Next we analyze each factor, respectively.
4.1 Task Modeling
4.1.1 Task Difficulty
Different from most existing works which assume that a worker
has the same quality for answering different tasks, some recent
works [53, 35] model the difficulty in each task. They assume that
each task has its difficulty level, and the more difficult a task is,
the harder a worker can correctly answer the task. For example,
in [53], it models the probability that worker w correctly answers
task t
i
as follows: Pr(v
w
i
= v
∗
i
| d
i
, q
w
) = 1/(1 + e
−d
i
·q
w
),
where d
i
∈ (0, +∞) represents the difficulty for task t
i
, and the
higher d
i
is, the easier task t
i
is. Intuitively, for a fixed worker
quality q
w
> 0, an easier task (high value of d
i
) leads to a higher
probability that the worker correctly answers the task.
4.1.2 Latent Topics
Different from modeling each task as a value (e.g., difficulty),
some recent works [19, 35, 57, 51] model each task as a vector
with K values. The basic idea is to exploit the diverse topics in a
task, where the topic number (i.e., K) is pre-defined. For example,
existing studies [19, 35] make use of the text description in each
task and adopt topic model techniques [6, 56] to generate a vector
of size K for the task; while Multi [51] learns a K-size vector with-
out referring to external information (e.g., text descriptions). Based
on the task models, a worker is probable to answer a task correctly
if the worker has high qualities on the task’s related topics.
4.2 Worker Modeling
4.2.1 Worker Probability
Worker probability uses a single real number (between 0 and 1)
to model a worker w’s quality q
w
∈ [0, 1], which represents the
ability that worker w correctly answers a task. The higher q
w
is, the
worker w has higher ability to correctly answer tasks. The model
543
Table 4: Comparisons of Different Methods that Address Truth Inference Problem in Crowdsourcing.
Method Task Types Task Modeling Worker Modeling Techniques
MV Decision-Making, Single-Choice No Model No Model Direct Computation
ZC [16] Decision-Making, Single-Choice No Model Worker Probability Probabilistic Graphical Model
GLAD [53] Decision-Making, Single-Choice Task Difficulty Worker Probability Probabilistic Graphical Model
D&S [15] Decision-Making, Single-Choice No Model Confusion Matrix Probabilistic Graphical Model
Minimax [61] Decision-Making, Single-Choice No Model Diverse Skills Optimization
BCC [27] Decision-Making, Single-Choice No Model Confusion Matrix Probabilistic Graphical Model
CBCC [46] Decision-Making, Single-Choice No Model Confusion Matrix Probabilistic Graphical Model
LFC [41] Decision-Making, Single-Choice No Model Confusion Matrix Probabilistic Graphical Model
CATD [30] Decision-Making, Single-Choice, Numeric No Model Worker Probability, Confidence Optimization
PM [5, 31] Decision-Making, Single-Choice, Numeric No Model Worker Probability Optimization
Multi [51] Decision-Making Latent Topics Diverse Skills, Worker Bias, Worker Variance Probabilistic Graphical Model
KOS [26] Decision-Making No Model Worker Probability Probabilistic Graphical Model
VI-BP [33] Decision-Making No Model Confusion Matrix Probabilistic Graphical Model
VI-MF [33] Decision-Making No Model Confusion Matrix Probabilistic Graphical Model
LFC N [41] Numeric No Model Worker Variance Probabilistic Graphical Model
Mean Numeric No Model No Model Direct Computation
Median Numeric No Model No Model Direct Computation
has been widely used in existing works [16, 26, 33, 5]. Some recent
works [53, 31] extend the worker probability to model a worker’s
quality in a wider range, e.g., q
w
∈ (−∞, +∞), and a higher q
w
means the worker w’s higher quality in answering tasks.
4.2.2 Confusion Matrix
Confusion matrix [15, 41, 33, 27, 46] is used to model a worker’s
quality for answering single-choice tasks. Suppose each task in T
has ` fixed choices, then the confusion matrix q
w
is an `× ` matrix,
where the j-th (1 ≤ j ≤ `) row, i.e., q
w
j,·
= [ q
w
j,1
, q
w
j,2
, . . . , q
w
j,`
],
represents the probability distribution of worker w’s possible an-
swers for a task if the truth of the task is the j-th choice. Each
element q
w
j,k
(1 ≤ j ≤ `, 1 ≤ k ≤ `) means that “given the truth of
a task is the j-th choice, the probability that worker w selects the
k-th choice”, i.e., q
w
j,k
= Pr(v
w
i
= k | v
∗
i
= j) for any t
i
∈ T .
For example, decision-making tasks ask workers to select ‘T’ (1st
choice) or ‘F’ (2nd choice) for each claim (` = 2), then an exam-
ple confusion matrix for w is q
w
=
h
0.8 0.2
0.3 0.7
i
, where q
w
1,2
= 0.2
means that if the truth of a task is ‘T’, the probability that the worker
answers the task as ‘F’ is 0.2.
4.2.3 Worker Bias and Worker Variance
Worker bias and variance [51, 41] are proposed to handle nu-
meric tasks, where worker bias captures the effect that a worker
may underestimate (or overestimate) the truth of a task, and worker
variance captures the variation of errors around the bias. For ex-
ample, given a set of photos with humans, each numeric task asks
workers to estimate the height of the human on it. Suppose a worker
w is modeled with bias τ
w
and variance σ
w
, then the answer v
w
i
given by worker w is modeled to draw from the Gaussian distri-
bution: v
w
i
∼ N (v
∗
i
+ τ
w
, σ
w
), that is, (1) a worker with bias
τ
w
0 (τ
w
0) will overestimate (underestimate) the height,
while τ
w
→ 0 leads to more accurate estimation; (2) a worker with
variance σ
w
0 means a large variation of error, while σ
w
→ 0
leads to a small variation of error.
4.2.4 Confidence
Existing works [30, 25] observe that if a worker answers plenty
of tasks, then the estimated quality for the worker is confident; oth-
erwise, if a worker answers only a few tasks, then the estimated
quality is not confident. Inspired by this observation, [35] assigns
higher qualities to the workers who answer plenty of tasks, than
the workers who answer a few tasks. To be specific, for a worker
w, it uses the Chi-Square distribution [3] with 95% confidence in-
terval, i.e., X
2
(0.975,|T
w
|)
as a coefficient to scale up the worker’s
quality, where |T
w
| is the number of tasks that worker w has an-
swered. X
2
(0.975,|T
w
|)
increases with |T
w
|, i.e., the more tasks w
has answered, the higher worker w’s quality is scaled to.
4.2.5 Diverse Skills
A worker may have various levels of expertise for different top-
ics. For example, a sports fan that rarely pays attention to enter-
tainment may answer tasks related to sports more correctly than
tasks related to entertainment. Different from most of the above
models which have an assumption that a worker has the same qual-
ity to answer different tasks, existing works [19, 35, 61, 51, 57,
59] model the diverse skills in a worker and capture a worker’s di-
verse qualities for different tasks. The basic ideas of [19, 61] are
that they model a worker w’s quality as a vector of size n, i.e.,
q
w
= [ q
w
1
, q
w
2
, . . . , q
w
n
], where q
w
i
indicates worker w’s quality
for task t
i
. Different from [19, 61], some recent works [35, 51,
57, 59] model a worker’s quality for different latent topics, i.e.,
q
w
= [ q
w
1
, q
w
2
, . . . , q
w
K
], where the number K is pre-defined, in-
dicating the number of latent topics. They [35, 51, 57, 59] assume
that each task is related to one or more topics in these K latent top-
ics, and a worker is highly probable to correctly answer a task if the
worker has a high quality in the task’s related topics.
5. TRUTH INFERENCE ALGORITHMS
Existing works [61, 19, 30, 5, 34, 16, 15, 53, 51, 41, 26, 33, 35,
27, 46] usually adopt the framework in Algorithm 1. Based on the
used techniques, they can be classified into the following three cat-
egories: direct computation [20, 39], optimization methods [61, 19,
30, 5] and probabilistic graphical model methods [34, 16, 15, 53,
51, 41, 26, 33, 35, 27, 46]. Next we talk about them, respectively.
5.1 Direct Computation
Some baseline methods directly estimate v
∗
i
(1 ≤ i ≤ n) based
on V , without modeling each worker or task. For decision-making
and single-label tasks, Majority Voting (MV) regards the truth of
each task as the answer given by most workers; while for numeric
tasks, Mean and Median are two baseline methods that regard the
mean and median of workers’ answers as the truth for each task.
5.2 Optimization
The basic idea of optimization methods is to set a self-defined
optimization function that captures the relations between workers’
qualities and tasks’ truth, and then derive an iterative method to
compute these two sets of parameters collectively. The differences
among existing works [5, 31, 30, 61] are that they model workers’
qualities differently and apply different optimization functions to
capture the relations between the two sets of parameters.
(1) Worker Probability. PM [5, 31] models each worker’s quality
as a single value, and the optimization function is defined as:
544
min
{q
w
},{v
∗
i
}
f({q
w
}, {v
∗
i
}) =
X
w∈W
q
w
·
X
t
i
∈T
w
d(v
w
i
, v
∗
i
),
where {q
w
} represents the set of all workers’ qualities, and simi-
larly {v
∗
i
} represents the set of all truth. It models a worker w’s
quality as q
w
≥ 0, and d(v
w
i
, v
∗
i
) ≥ 0 defines the distance be-
tween worker’s answer v
w
i
and the truth v
∗
i
: the similar v
w
i
is to
v
∗
i
, the lower the value of d(v
w
i
, v
∗
i
) is. Intuitively, to minimize
f({q
w
}, {v
∗
i
}), a worker w’s high quality q
w
corresponds to a low
value in d(v
∗
i
, v
w
i
), i.e., worker w’s answer should be close to the
truth. By capturing the intuitions, similar to Algorithm 1, PM [5,
31] develops an iterative approach, and in each iteration, it adopts
the two steps as illustrated in Section 3.
(2) Worker Probability and Confidence. Different from above,
CATD [30] considers both worker probability and confidence in
modeling a worker’s quality. As discussed in Section 4.2.4, each
worker w’s quality is scaled up to a coefficient of X
2
(0.975,|T
w
|)
,
i.e., the more tasks w has answered, the higher worker w’s quality
is scaled to. It develops an objective function, with the intuitions
that a worker w who gives answers close to the truth and answers
a plenty of tasks should have a high quality q
w
. Similarly it adopts
an iterative approach, and iterates the two steps until convergence.
(3) Diverse Skills. Minimax [61] leverages the idea of minimax en-
tropy [63]. To be specific, it models the diverse skills of a worker
w across different tasks and focuses on single-label tasks (with `
choices). It assumes that for a task t
i
, the answers given by w are
generated by a probability distribution π
w
i,·
= [ π
w
i,1
, π
w
i,2
, . . . , π
w
i,`
],
where each π
w
i,j
is the probability that worker w answers task t
i
with the j-th choice. Following this, an objective function is de-
fined by considering two constraints for tasks and workers: for a
task t
i
, the number of answers collected for a choice equals the sum
of corresponding generated probabilities; for a worker w, among all
tasks answered by w, given the truth is the j-th choice, the number
of answers collected for the k-th choice equals the sum of corre-
sponding generated probabilities. Finally [61] devises an iterative
approach to infer the two sets of parameters {v
∗
i
} and {π
w
}.
5.3 Probabilistic Graphical Model (PGM)
A probabilistic graphical model (PGM) [28] is a graph which ex-
presses the conditional dependency structure (represented by edges)
between random variables (represented by nodes). Figure 1 shows
the general PGM adopted in existing works. Each node represents
a variable. There are two plates, respectively for workers and tasks,
where each one represents the repeating variables. For example, the
plate for workers represents |W| repeating variables, where each
variable corresponds to a worker w ∈ W. For the variables, α, β,
and v
w
i
are known (α and β are priors for q
w
and v
∗
i
, which can be
set based on the prior knowledge); q
w
and v
∗
i
are latent or unknown
variables, which are two desired variables to compute. The directed
edges model the conditional dependence between a child node and
its associated parent node(s) in the sense that the child node fol-
lows a probabilistic distribution conditioned on the values taken by
the parent node(s). For example, three conditional distributions in
Figure 1 are Pr(q
w
| α), Pr(v
∗
i
| β) and Pr(v
w
i
| q
w
, v
∗
i
).
Next we illustrate the details (optimization goal and the two steps)
of each method using PGM. In general the methods differ in the
used worker model. It can be classified into three categories: worker
probability [16, 53, 26, 33], confusion matrix [15, 41, 27, 46] and
diverse skills [19, 35, 51]. For each category, we first introduce its
basic method, e.g., ZC [16], and then summarize how other meth-
ods [53, 26, 33] extend the basic method ZC [16].
(1) Worker Probability: ZC [16] and its extensions [53, 26, 33].
workers
tasks
v
i
w
α
β
q
w
v
i
*
|W |
n
Figure 1: A General PGM (Probabilistic Graphical Model).
ZC [16] adopts a PGM similar to Figure 1, with the simplifica-
tion that it does not consider the priors (i.e., α, β). Suppose all
tasks are decision-making tasks (v
∗
i
∈ {T, F}) and each worker’s
quality is modeled as worker probability q
w
∈ [0, 1]. Then
Pr(v
w
i
| q
w
, v
∗
i
) = (q
w
)
1
{v
w
i
=v
∗
i
}
· (1 − q
w
)
1
{v
w
i
6=v
∗
i
}
,
which means that the probability worker w correctly (incorrectly)
answers a task is q
w
(1 − q
w
). For decision-making tasks, ZC [16]
tries to maximize the probability of the occurrence of workers’ an-
swers, called likelihood, i.e., max
{q
w
}
Pr(V | {q
w
}), which re-
gards {v
∗
i
} as latent variables:
Pr(V | {q
w
}) =
1
2
·
Y
n
i=1
X
z∈{T, F}
Y
w∈W
i
Pr(v
w
i
| q
w
, v
∗
i
= z). (1)
However, it is hard to optimize due to the non-convexity. Thus
ZC [16] applies the EM (Expectation-Maximization) framework [17]
and iteratively updates q
w
and v
∗
i
to approximate its optimal value.
Note ZC [16] develops a system to address entity linking for online
pages. In this paper we focus on the part of leveraging the crowd’s
answers to infer the truth (i.e., Section 4.3 in [16]), and we omit
other parts (e.g., constraints on its probabilistic model).
There are several extensions of ZC, e.g., GLAD [53], KOS [26],
VI-BP [33], VI-MF [33], and they focus on different perspectives:
Task Model. GLAD [53] extends ZC [16] in task model. Rather
than assuming that each task is the same, it [53] models each task
t
i
’s difficulty d
i
∈ (0, +∞) (the higher, the easier). Then it models
the worker’s answer as Pr(v
w
i
= v
∗
i
| d
i
, q
w
) = 1/(1 + e
−d
i
·q
w
),
and integrates it into Equation 1 to approximate the optimal value
using Gradient Descent [28] (an iterative method).
Optimization Function. KOS [26], VI-BP [33], and VI-MF [33]
extend ZC [16] in an optimization goal. Recall that ZC tries to
compute the optimal {q
w
} that maximizes Pr(V | {q
w
}), which is
the Point Estimate. Instead, [26, 33] leverage the Bayesian Estima-
tors to calculate the integral of all possible q
w
, and the target is to
estimate the truth v
∗
i
= argmax
z∈{T, F}
Pr(v
∗
i
= z | V ), where
Pr(v
∗
i
= z | V ) =
Z
{q
w
}
Pr(v
∗
i
= z, {q
w
} | V ) d{q
w
}. (2)
It is hard to directly compute Equation 2, and existing works [26,
33] seek for Variational Inference (VI) techniques [49] to approx-
imate the value: KOS [26] first leverages Belief Propagation (one
typical VI technique) to iteratively approximate the value in Equa-
tion 2, then [33] proposes a more general model based on KOS,
called VI-BP. Moreover, it [33] also applies Mean Field (anther VI
technique) in VI-MF to iteratively approach Equation 2.
(2) Confusion Matrix: D&S [15] and its extensions [41, 27, 46].
D&S [15] focuses on single-label tasks (with fixed ` choices)
and models each worker as a confusion matrix q
w
with size ` × `
(Section 4.2.2). The worker w’s answer follows the probability
Pr(v
w
i
| q
w
, v
∗
i
) = q
w
v
∗
i
,v
w
i
. Similar to Equation 1, D&S [15] tries
to optimize the function argmax
{q
w
}
Pr(V | {q
w
}), where
Pr(V | {q
w
}) =
Y
n
i=1
X
1≤z≤`
Pr(v
∗
i
= z) ·
Y
w∈W
i
q
w
z,v
w
i
,
and it applies the EM framework [17] to devise two iterative steps.
The above method D&S [15], which models a worker as a con-
fusion matrix, is also a widely used model. There are some exten-
sions, e.g., LFC [41], LFC N [41], BCC [27] and CBCC [46].
545