Large-scale Visual Sentiment Ontology
and Detectors Using Adjective Noun Pairs
Damian Borth
1
Rongrong Ji
2
Tao Chen
2
Thomas Breuel
1
Shih-Fu Chang
2
1
University of Kaiserslautern, Germany
2
Columbia University, USA
{d_borth, tmb}@cs.uni-kl.de {rrji, taochen, sfchang}@ee.columbia.edu
ABSTRACT
We address the challenge of sentiment analysis from visual
content. In contrast to existing methods which infer senti-
ment or emotion directly from visual low-level features, we
propose a novel approach based on understanding of the vi-
sual concepts that are strongly related to sentiments. Our
key contribution is two-fold: first, we present a method built
upon psychological theories and web mining to automatical-
ly construct a large-scale Visual Sentiment Ontology (VSO)
consisting of more than 3,000 Adjective Noun Pairs (AN-
P). Second, we propose SentiBank, a novel visual concept
detector library that can be used to detect the presence of
1,200 ANPs in an image. The VSO and SentiBank are dis-
tinct from existing work and will open a gate towards var-
ious applications enabled by automatic sentiment analysis.
Experiments on detecting sentiment of image tweets demon-
strate significant improvement in detection accuracy when
comparing the proposed SentiBank based predictors with
the text-based approaches. The effort also leads to a large
publicly available resource consisting of a visual sentiment
ontology, a large detector library, and the training/testing
benchmark for visual sentiment analysis.
Categories and Subject Descriptors
H.3.3 [Information Storage and Retrieval]: Information
Retrieval and Indexing
Keywords
Sentiment Prediction, Concept Detection, Ontology, Social
Multimedia
1. INTRODUCTION
Nowadays the Internet, as a major platform for communi-
cation and information exchange, provides a rich repository
of people’s opinion and sentiment about a vast spectrum of
topics. Such knowledge is embedded in multiple facets, such
as comments, tags, browsing actions, as well as shared media
objects. The analysis of such information either in the area
of opinion mining, affective computing or sentiment analysis
plays an important role in behavior sciences, which aims to
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
MM’13, October 21–25, 2013, Barcelona, Spain.
Copyright 2013 ACM 978-1-4503-2404-5/13/10 ...$15.00.
http://dx.doi.org/10.1145/2502081.2502282.
Figure 1: Tweets from the “2012 Year on Twitter” collec-
tion: Barack Obamas reelection tweet (left) and a tweet cap-
turing the destruction caused by Hurricane Sandy (right).
Both tweets are characterized by a short text (”four more
years” and ”rollercoaster at sea” respectively) and conveying
the sentiment visually.
understand and predict human decision making [26] and en-
ables applications such as brand monitoring, stock market
prediction, or political voting forecasts.
So far, the computational analysis of sentiment mostly
concentrates on the textual content [26]. Limited effort-
s have been conducted to analyze sentiments from visual
content such as images and videos, which is becoming a per-
vasive media type on the web. For example, two of the most
popular tweets in 2012 (see Fig. 1) conveying valuable sen-
timent information primarily visually
1
. Thus, an open issue
in sentiment analysis research is the need of visual content
analysis.
This problem poses a set of unique challenges as it ad-
dresses abstract human concepts in the sense of emotion
and affect. Typically, semantic concept detection in images
is concerned with the physical presence of objects or scenes
like “car” or “building”. On the other hand sentiment could
differ among persons as the stimuli evoked human responses
are naturally subjective. In some sense, this is analogous
to the differentiation between content-based image retrieval
(CBIR) and emotional semantic image retrieval (ESIR) [33].
There exists an affective gap in ESIR [21] between low-level
features and the emotional content of an image reflecting
a particular sentiment, similar to the well-known semantic
gap in CBIR between low-level features and image semantic-
s. To fill the semantic gap, mid-level representations based
on visual concepts have been proposed. In this paper, we
propose to discover and detect a set of visual concepts that
can be used to fill the affective gap and automatically infer
the sentiments reflected in an image. Note, that our mid-
1
Please note that, throughout the paper we will define
sentiment similarly to [26], as the polarity of an opinion
item which either can be positive, neutral or negative
Sentiment
Prediction
Detector
Training and
Validation
Adj + Nouns
= ANP’s
Data-driven
Discovery
Wheel of
Emotion
(Psychology)
24 emotions
Sentiment
Words
Visual Sentiment
Ontology
SentiBank
(
1200 detectors)
Figure 2: Overview of the proposed framework for constructing the visual sentiment ontology and SentiBank. Applications
in multimodal sentiment prediction is also shown.
level representation is much expressive than the ones (e.g.,
color schemes or geometric shapes) described in [33] and
has a better capability for explaining sentiment prediction
results.
We apply the psychological theory, Plutchik’s Wheel of
Emotions [27], as the guiding principle to construct a large-
scale visual sentiment ontology (VSO) that consists of
more than 3,000 semantic concepts. Our construction cri-
teria ensure that each selected concept (1) reflects a strong
sentiment, (2) has a link to emotions, (3)is frequently used
in practice, and (4) has a reasonable detection accuracy.
To satisfy these conditions, we introduce Adjective Noun
Pairs (ANP) such as“beautiful flower”or“disgusting food”.
The advantage of using ANPs, compared to nouns or adjec-
tives only, is its capability to turn a neutral noun like “dog”
into an ANP with strong sentiment like“cute dog” by adding
an adjective with a strong sentiment. Such combined phras-
es also make the concepts more detectable than adjectives
(like “beautiful”), which are typically abstract and difficult
to detect. Building upon the VSO we introduce SentiBank,
a library of trained concept detectors providing a mid-level
visual representation. We show - through extensive exper-
iments - that useful detector performance can be achieved
for 1,200 ANP concepts, which form the released detector li-
brary SentiBank. Further, we demonstrate the usefulness of
the proposed approach towards sentiment prediction on im-
age tweets as it outperforms text-based prediction approach-
es by a very large margin. In summary, our contributions are
- first, a systematic, data-driven methodology for the con-
struction of a visual sentiment ontology from user-generated
content and folksonomies on the web; second, the large-
scale Visual Sentiment Ontology founded by a well-known
psychological model; third, a mid-level representation built
on automatic detectors of the discovered concepts in order
to bridge the affective gap mentioned earlier; and, forth, the
public release of the VSO including its large-scale dataset,
the SentiBank detector library, and the benchmark for visual
sentiment analysis.
In the rest of the paper, we first discuss related work
(Sec.2) and show an overview of the proposed framework
(Sec. 3). Then, the design and construction methodology
of the VSO (Sec.4) and SentiBank, the proposed mid-level
visual concept representation (Sec.5) are discussed. Finally,
application in image tweet sentiment prediction is presented
(Sec.6).
2. RELATED WORK
The challenge of automatically detecting semantic con-
cepts such as objects, locations, and activities in visual da-
ta, referred to as video annotation [1], concept detection
[28], semantic indexing [25] or multimedia event detection
[20], has been studied extensively over the last decade. In
benchmarks like TRECVID [25] or the PASCAL visual ob-
ject challenge [10], the research community has investigated
a variety of features and statistical models. In addition,
there also has been much work in creating large ontologies
and datasets [7, 14, 29]. Typically, such vocabularies are
defined according to utility for retrieval, coverage, diversity,
availability of training material, and its detectability by au-
tomatic detection systems [23, 25]. Recent approaches have
also turned towards web portals like Flickr and YouTube
as information sources for visual learning, employing user-
generated tags as an alternative to manual labels [16, 31].
Aligned with the aforementioned trend, our approach also
exploits large-scale image tags available on the web. Our fo-
cus, however, is less on concept detection itself but rather on
the construction of an ontology of visually detectable ANPs
serving as mid-level representation of sentiment attributes of
visual content. In contrast, the prior works focus on phys-
ical concepts corresponding to objects, scenes, location but
not concepts that characterize sentiment visually.
With respect to sentiment analysis, much progress has
been achieved in text analysis [9, 30] and textual dictionary
creation [9, 35]. However, efforts for visual analysis fall far
behind. The closest that comes to sentiment analysis for
visual content is the analysis of aesthetics [6, 15, 22], in-
terestingness [12], and affect or emotions [13, 21, 37, 36].
To this end, either low-level features are directly taken to
predict emotion [18, 13], or indirectly by facial expressions
detection [32] or user intent [11]. Similarly [34], which intro-
duced a high-level representation of emotions, is limited to
low-level features such as color based schemes. Please refer
to [15, 33] for a comprehensive study of aesthetics and emo-
tions in images. Compared to the above works, our proposed
approach is novel and ambitious in two ways. First, we build
a large-scale ontology of semantic concepts correlated with
strong sentiments like “beautiful landscape” or “dark clouds”
as a complement to a textual sentiment dictionary [9, 35].
Such an ontology is the first of its kind and would open
new research opportunities for the multimedia community
and beyond. Second, from such an ontology and a publicly
shared detector library a mid-level visual representation can
be learned for the purpose of robust sentiment prediction.
Only a few small datasets exist today for affect / emo-
tion analysis on visual content. A prominent one is the
International Affective Picture System (IAPS) [17] provid-
ing normative ratings of emotion (pleasure, arousal, domi-
nance) for a set of color photographs. The dataset consists
of 369 photos covering various scenes showing insects, pup-
pies, children, poverty, diseases and portraits, which are rat-
ed by 60 participants using affective words. Similarly, the
Geneva Affective Picture Database (GAPED) [4] provides
Figure 3: Plutchik’s Wheel of Emotions and the visualiza-
tion interface of the ontology based on the wheel.
730 pictures including negative (e.g., spiders, snakes, scenes
containing human rights violation), positive (e.g., human
and animal babies, nature sceneries) and neutral pictures.
All pictures were rated according to valence, arousal, and
the consistency of the represented scenes. In [21], the Af-
fective Image Classification Dataset includes two separate
datasets of abstract painting (228 paintings) and artistic
photos (807 photos), which are labeled with 8 basic emo-
tions through a crowd-sourcing procedure. In contrast to
the above mentioned datasets our work provides a signifi-
cantly larger dataset (about 0.5 million) of images crawled
from social media and labeled with thousands of ANP con-
cepts. In addition, we created a separate image dataset from
Twitter for a sentiment prediction benchmark.
3. FRAMEWORK OVERVIEW
An overview of the proposed framework is shown in Fig. 2.
The construction process is founded on psychological prin-
ciples such as Plutchik’s Wheel of Emotions [27]. During
the first step, we use each of the 24 emotions defined in
Plutchik’s theory to derive search keywords and retrieve
images and videos from Flickr and YouTube. Tags asso-
ciated with the retrieved images and videos are extracted -
for example “joy” leads to “happy”, “beautiful”, and “flow-
er”. These tags are then analyzed to assign sentiment values
and to identify adjectives, verbs, and nouns. The set of al-
l adjectives with strong sentiment values and all nouns is
then used to form adjective noun combinations or Adjec-
tive Noun Pairs (ANP) such as ”beautiful flowers” or
”sad eyes”. Those ANPs are then ranked by their frequency
on Flickr and sampled to form a diverse and comprehensive
ontology containing more than 3,000 ANP concepts. We
then train individual detectors using Flickr images that are
tagged with an ANP and keep only detectors with reason-
able performance to form SentiBank. This detector library
consists of 1,200 ANP concept detectors providing a 1,200
dimension ANP detector response for a given image. As a
sample application, we apply SentiBank and train classifiers
to predict sentiment values of image tweets and demonstrate
a superior performance over conventional sentiment predic-
tion using text only.
4. VISUAL SENTIMENT ONTOLOGY
In this section we outline the design and systematic con-
disgusting
gross
food
nasty
sick
dirty
deaddead
face
blood
insect
amazing
beautiful
nature
wonder
light
love
skysky
eyes
clouds
landscape
terror
horror
zombie
fear
dark
street
halloweenhalloween
war
undead
bomb
joy
happy
love
smile
beautiful
flowers
lightlight
nature
kids
christmas
joy terror amazement
disgust
Figure 4: Example top tags for different emotions. Colors
of the boxes (green, grey, red) indicate different sentiments
(positive, neutral, negative).
struction of the proposed Visual Sentiment Ontology (VSO).
Here we focus on sentiment or emotion expressed by the
content owner shared on social media such as Twitter. We
assume the sentiments of the receivers (i.e., viewers of the
visual content), though not directly addressed in this paper,
are strongly related to those of the content owners. Our goal
is to construct a large-scale ontology of semantic concepts,
which (1) reflect a strong sentiment, (2) have a link to an
emotion, (3) are frequently used and (4) have reasonable
detection accuracy. Additionally, the VSO is intended to be
comprehensive and diverse enough to cover a broad range
of different concept classes such as people, animals, objects,
natural or man-made places, and so on.
4.1 Psychological Foundation
To establish a solid foundation for the construction of the
VSO we utilize a well-known emotion model derived from
psychological studies. There are several well-known early
works such as Darwin’s evolutionary motivation of emotion-
s [5], Ekman’s facial expression system [8] and Osgood’s ap-
praisal and valence model [24]. Here, focus on Plutchnik’s
Wheel of Emotions [27] as seen in Fig. 3 is organized into
8 basic emotions, each with 3 valences. Beginning from the
top we have:
1. ecstasy → joy → serenity
2. admiration → trust → acceptance
3. terror → fear → apprehension
4. amazement → surprise → distraction
5. grief → sadness → pensiveness
6. loathing → disgust → boredom
7. rage → anger → annoyance
8. vigilance → anticipation → interest
Why Plutchnik’s Emotion Model? The model is in-
spired by chromatics in which emotions elements are ar-
ranged along a wheel and bi-polar emotions are opposite
to each other - a useful property for the construction of a
sentiment ontology. Further, it maps well to psychologi-
cal theories such as Ekman, where 5 basic emotions are the
same (anger, disgust, fear, sadness, surprise) while Ekman’s
“happiness” maps well to Plutchnik’s “joy”. Compared to
the emotional model utilized in [21], Plutchnik basic emo-
tions correspond to all 4 negative emotions but have slightly
different positive emotions. In contrast, Plutchnik intro-
duced two additional basic emotions (interest, trust) and
organizes each of them into 3 intensities providing a richer
set of different emotional valences. Statistics of our crawled
Table 1: Statistics of the Visual Sentiment Ontology con-
struction process
(a) Flickr YouTube
# of emotions 24 24
images or videos 150,034 166,342
tags 3,138,795 3,079,526
distinct top 100 tags 1,146 1,047
(b) Sentiment Words
distinct top 100 tags 1,771
pos+neg adjectives 268
neutral adjectives 0
total adjectives 268
pos+neg nouns 576
neutral nouns 611
total nouns 1,187
(c) VSO Statistics
ANP concept candidates 320k
ANPs (with non-empty images) 47k
ANPs included in VSO 3k
top pos. adjectives beautiful, amazing, cute
top neg. adjectives sad, angry, dark
top nouns face, eyes, sky
dataset confirm useful contribution of each emotion group
in Plutchik to the final VSO.
4.2 Sentiment Word Discovery
Initial Image & Video Retrieval: For each of the 24
emotions we retrieve images and videos from Flickr and Y-
ouTube respectively and then extract their distinct associ-
ated tags by the Lookapp tool [2]. In total we retrieve about
310k images and videos and about 6M tags, which are made
of a set of 55k distinct tags. An overview of this step can be
seen in Tab. 1 (a).
Tags Analysis: For tag analysis we first remove stop-
words and perform stemming. For each emotion, we perfor-
m tag frequency analysis to obtain the top 100 tags. Ex-
amples of such top tags can be seen in Fig. 4. Finally, the
sentiment value of each tag is computed using two popular
linguistics based sentiment models, SentiWordNet [9] and
SentiStrength [30]. In this work, each word is assigned a
sentiment value ranging from -1 (negative) to +1 (positive).
Overall, as shown Tab. 1 (b), we are able to retrieve 1146
distinct tags from Flickr and 1,047 distinct tags from Y-
ouTube forming the final set of 1,771 distinct tags with 1,187
nouns (576 positive and negative ones and 611 neutral ones)
and 268 positive or negative adjectives. Note that we ig-
nore verbs in this work because of the current focus on still
images.
4.3 Adjective Noun Pair (ANP) Construction
Looking closer at the results of the previous step we can
see that the 576 discovered nouns with positive or negative
sentiment would satisfy the first concept selection condition
mentioned above for ontology construction (reflecting strong
sentiment), but in this case we would not be able to include
the 611 neutral nouns. As for the adjectives, all 268 have
either a positive or negative sentiment value (satisfying con-
dition (1)) but probably we would not be able to satisfy con-
dition (4): reasonable detection accuracy. Visual learning
Figure 6: Top: Count of images on Flickr per ANP. Bot-
tom: count of CC images downloaded per ANP (limited to
max of 1000 images per ANP).
of adjectives is understandably difficult due to its abstract
nature and high variability. Therefore, we propose adjective
nouns combinations or Adjective Noun Pairs (ANP) to
be the main semantic concept elements of the VSO. The ad-
vantage of using ANPs, as compared to nouns or adjectives
only, is the feasibility of turning a neutral noun into a strong
sentiment ANP. Such combined concepts also make the con-
cepts more detectable, compared to adjectives only. The
above described ANP structure shares certain similarity to
the recent trend in computer vision and multimedia concept
detection, i.e. bi-concepts [19] or TRECVID’s concept pairs
[25].
Candidate Selection: The set of all strong sentiment
value adjectives and the set of all nouns are now used to
form ANPs such as “beautiful flower” or “disgusting food”.
After ANP concepts are formed, an extra text analysis step
is employed to avoid ANPs that correspond to named en-
tities with meaning changed (e.g., “hot” + “dog” leads to
a named entity instead of a generic concept). Obviously,
during the construction of ANPs we also have to fuse the
sentiment value of the adjective and the noun. This is done
by applying a simple model to sum up the corresponding
sentiment values s(AN P ) = s(adj) + s(noun) where the
sentiment value s(ANP ) is between -2 and +2. Obvious-
ly, with this model we have to be careful with cases like
“abused” being negative and “child” being positive forming
the ANP “abused child”, which reflects definitely a strong
negative sentiment. We address this issue, by identifying
ANPs that include an adjective and a noun with opposite
sentiment values. We observed that in such cases the ad-
jective usually has a stronger impact on the overall ANP
sentiment than the noun and thus let the ANP inherits the
sentiment value of the adjective.
Candidate Ranking: Those ANPs candidates (about
320k) are then ranked by their frequency on Flickr to re-
move meaningless or extremely rare constructions like e.g.
“frightened hat” or “happy happiness”. Having this ranked
list of ANP frequencies (characterized by a long tail as seen
in Fig. 6 (top)), we dismiss all ANPs with no images found
on Flickr. This leads to a remaining set 47k ANP candi-
dates. In this step we also eliminate cases where both, the
singular and plural forms of an ANPs exists in the VSO. In
such a case we take the more frequent one.
Figure 5: Left: Selected images for four sample ANPs, (a),(c) reflecting a positive sentiment and (b), (d), a negative one.
Right: top detected images by SentiBank ANPs with high detection accuracy (top) and low accuracy (bottom). Correct
detections are surrounded by green and thick frames and incorrect ones by red and dashed frames. Faces in the images are
blurred.
Table 2: Top 3 ANPs for basic emotions.
Emotion Top ANPs
joy happy smile, innocent smile, happy christmas
trust christian faith, rich history, nutritious food
fear dangerous road, scary spider, scary ghost
surprise pleasant surprise, nice surprise, precious gift
sadness sad goodbye, sad scene, sad eyes
disgust nasty bugs, dirty feet, ugly bug
anger angry bull, angry chicken, angry eyes
anticipation magical garden, tame bird, curious bird
Ontology Sampling: The final step is to subsample the
concepts in a diverse and balanced way and include those
with a high frequency only. To avoid dominance by just a
few popular adjectives, we partition candidate concepts into
individual adjective sets and sample from each adjective a
subset of ANPs. Further we only take ANPs if they have
sufficient (currently > 125) images found on Flickr.
Link back to Emotions: We are interested in how the
discovered ANPs are related to the basic emotions used in
the very first retrieval step. Here, we measure the counts of
images that have both the emotion term and the ANP string
in their meta-data and normalize the resulting 24 dimension
histogram to unit sum. This way a two-directional connec-
tion between an emotion and an ANP can be established.
For example, the most dominant emotion for “happy smile”
ANP is“joy”and for the emotion“disgust” the popular ANP
is “nasty bugs”. More examples can be seen in Table 2.
The final VSO contains more than 3,000 ANP concepts
with 268 adjectives and their corresponding ANPs. Some of
top ranked APNs are: “happy birthday”, “beautiful flower”,
and“little girl”being positive and“dark night”, “heavy rain”,
and “broken window” being the negative counterpart.
4.4 Flickr CC Dataset & Visualization Tool
An essential part of the VSO is its image dataset repre-
senting each ANP. The images are used for SentiBank de-
tector training (Sec. 5). We used the Flickr API to retrieve
and download Creative Common (CC) images for each AN-
P (limited to 1000 images by the API service) and include
only images that contain the ANP string either in the title,
tag or description of the image. With this we were able to
download a sufficient amount of CC images for 1,553 of the
3,000 ANPs (in total about 500k images). The distribution
of image count per ANP can be seen in Fig. 6 (bottom).
Selected images of four example ANPs are show in Fig. 5
(left).
To help visualize the VSO and the associated dataset,
we have developed two novel visualization techniques, one
based on Wheel of Emotion (shown in Fig. 3) and the other
the well-known TreeMap hierarchical visualization method
(Fig. 8). The Emotion Wheel interface allows users to view
and interact with the Plutchik 24 emotions directly and then
zoom in to explore specific ANP concepts and associated im-
ages. The TreeMap interface offers a complementary way of
navigating through different levels of VSO - emotion, ad-
jective, noun, and ANPs. At each level, the map shows s-
tatistics and summaries of information from the level below.
Interactive demos of these tools are available online
2
.
5. SENTIBANK
Given the Visual Sentiment Ontology constructed above,
we propose SentiBank, a novel visual sentiment analysis
framework using the output of ANP detectors as a mid-
level concept representation for each image. Its objective is
to detect ANP concept presence and to characterize the sen-
timent reflected in visual content. To this end, we address
several key issues, namely ANP label reliability, design of
individual ANP detectors, detector performance, and cover-
age.
5.1 Reliability of ANP labels
It is known that web labels may not be reliable [7, 31].
Using Flickr tags directly as pseudo labels of ANPs might
incur either false positive, i.e. an image is labeled by an
ANP but actually does not show the ANP, or false negative,
i.e. if an image is not labeled with an ANP it does not imply
the ANP is not present in the image.
Dealing with Pseudo Labels: Considering the poten-
tial of false positive, we further evaluate the ANP labels by
an Amazon Mechanical Turk (AMT) experiment. We ran-
2
http://visual-sentiment-ontology.appspot.com/
