High level describable attributes for predicting aesthetics and interestingness

TLDR: This paper demonstrates a simple, yet powerful method to automatically select high aesthetic quality images from large image collections and demonstrates that an aesthetics classifier trained on describable attributes can provide a significant improvement over baseline methods for predicting human quality judgments.

Abstract: With the rise in popularity of digital cameras, the amount of visual data available on the web is growing exponentially. Some of these pictures are extremely beautiful and aesthetically pleasing, but the vast majority are uninteresting or of low quality. This paper demonstrates a simple, yet powerful method to automatically select high aesthetic quality images from large image collections. Our aesthetic quality estimation method explicitly predicts some of the possible image cues that a human might use to evaluate an image and then uses them in a discriminative approach. These cues or high level describable image attributes fall into three broad types: 1) compositional attributes related to image layout or configuration, 2) content attributes related to the objects or scene types depicted, and 3) sky-illumination attributes related to the natural lighting conditions. We demonstrate that an aesthetics classifier trained on these describable attributes can provide a significant improvement over baseline methods for predicting human quality judgments. We also demonstrate our method for predicting the “interestingness” of Flickr photos, and introduce a novel problem of estimating query specific “interestingness”.

Figures

Figure 4.6: Weights w(t, i) for every term (t, i) sorted in descending order to show the decay behavior of the weights.

Figure 4.1: Comparative Analysis of Decision-tree based SVM and non-Decision-tree based SVMs for classification. For the Decision-tree based approach, we consider 2 binary attributes, ‘Presence of people’ and ‘Presence of Faces/Portraits’ which lead to 4 possible states and hence 4 possible SVMs.

Figure 4.12: Attributes corresponding to highest absolute values of the means for the DPChallenge Dataset

Figure 3.2: DPChallenge photos ranked by aesthetics. This figure shows the bottom 36 images as ranked by our aesthetics classifier.

Figure 3.1: DPChallenge photos ranked by aesthetics. This figure shows the top 78 images as ranked by our aesthetics classifier.

Figure 4.2: Classification using the RBF kernel, K(X,Z) = e−|X−Z| 2 , Accuracy (Combined features) = 80.375% (1286/1600), Accuracy (Low Level features) = 68.5625 % (1097/1600), Accuracy (High Level features) = 74.6875% (1195/1600)

Full text

S
S
S
t
t
t
o
o
o
n
n
n
y
y
y
B
B
B
r
r
r
o
o
o
o
o
o
k
k
k
U
U
U
n
n
n
i
i
i
v
v
v
e
e
e
r
r
r
s
s
s
i
i
i
t
t
t
y
y
y
The official electronic file of this thesis or dissertation is maintained by the University
Libraries on behalf of The Graduate School at Stony Brook University.
©
©
©
A
A
A
l
l
l
l
l
l
R
R
R
i
i
i
g
g
g
h
h
h
t
t
t
s
s
s
R
R
R
e
e
e
s
s
s
e
e
e
r
r
r
v
v
v
e
e
e
d
d
d
b
b
b
y
y
y
A
A
A
u
u
u
t
t
t
h
h
h
o
o
o
r
r
r
.
.
.

High Level Describable Attributes for Predicting Aesthetics and
Interestingness
A Thesis Presented
by
Sagnik Dhar
to
The Graduate School
in Partial Fulfillment of the
Requirements
for the Degree of
Master of Science
in
Computer Science
Stony Brook University
December 2010

Stony Brook University
The Graduate School
Sagnik Dhar
We, the thesis committee for the above candidate for the
Master of Science degree,
hereby recommend acceptance of this thesis.
Professor Tamara Berg - Advisor
Assistant Professor, Department of Computer Science
Professor Dimitris Samaras - Chairperson of Defense
Associate Professor, Department of Computer Science
Professor Alexander Berg
Assistant Professor, Department of Computer Science
This thesis is accepted by the Graduate School
Lawrence Martin
Dean of the Graduate School
ii

Abstract of the Thesis
High Level Describable Attributes for Predicting Aesthetics and
Interestingness
by
Sagnik Dhar
Master of Science
in
Computer Science
Stony Brook University
2010
With the rise in popularity of digital cameras, the amount of visual data available
on the web is growing exponentially. Some of these pictures are extremely beautiful
and aesthetically pleasing. Unfortunately the vast majority are uninteresting or of
low quality. This paper demonstrates a simple, yet powerful method to automatically
select high aesthetic quality images from large image collections with performance
significantly better than the state of the art. We also show significantly better results
on predicting the interestingness of Flickr images, and on a novel problem of predicting
query specific interestingness. Our aesthetic quality estimation method explicitly
predicts some of the possible image cues that a human might use to evaluate an
image and then uses them in a discriminative approach. These cues or high level
describable image attributes fall into three broad types: 1) compositional attributes
related to image layout or configuration, 2) content attributes related to the objects or
scene types depicted, and 3) sky-illumination attributes related to the natural lighting
conditions. We demonstrate that an aesthetics classifier trained on these describable
attributes can provide a significant improvement over state of the art methods for
predicting human quality judgments.
iii

Dedicated to my mother, whose forays into the world of Architecture instilled in me
my earliest notions of aesthetics.

Frequently asked questions

Q1. What are the contributions in this paper?

This paper demonstrates a simple, yet powerful method to automatically select high aesthetic quality images from large image collections with performance significantly better than the state of the art. The authors also show significantly better results on predicting the interestingness of Flickr images, and on a novel problem of predicting query specific interestingness. Their aesthetic quality estimation method explicitly predicts some of the possible image cues that a human might use to evaluate an image and then uses them in a discriminative approach. The authors demonstrate that an aesthetics classifier trained on these describable attributes can provide a significant improvement over state of the art methods for predicting human quality judgments. 

Q2. What are the future works in this paper?

In the future, the authors plan to expand their set of attributes to extract other describable image features, and to apply these attributes to related tasks such as image emotion estimation. The authors also plan to more thoroughly explore ideas of query specific interestingness, including methods for query specific attribute selection, and methods for interestingness transfer. The following applications could potentially be very useful: 43 1. Classification of aesthetic quality is done using a measure of how much a test image deviates from the ideal image. The measure of deviation of the test image from the ideal image could be used to suggest potential changes to the user/photographer to help improve the aesthetic quality of his photograph. 

Q3. What is the title of the paper?

264.1 Decision-Tree based SVMs . . . . . . . . . . . . . . . . . . . . . . . . 29 4.2 Radial-Basis-Function Kernel & Polynomial Kernel results . . . . . . 30 4.3 Attribute contribution for classification . . . . . . . . . . . . . . . . . 32vBibliography 46vi 

Q4. What are the three types of images that can be used to predict the quality of images?

These cues or high level describable image attributes fall into three broad types: 1) compositional attributes related to image layout or configuration, 2) content attributes related to the objects or scene types depicted, and 3) sky-illumination attributes related to the natural lighting conditions.