SUBMITTED FOR PUBLICATION AT IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 1
Estimating the Helpfulness and Economic Impact of
Product Reviews: Mining Text and Reviewer
Characteristics
Anindya Ghose, Panagiotis G. Ipeirotis, Member, IEEE,
Abstract
—With the rapid growth of the Internet, the ability of users to create and publish content has created active electronic communities that
provide a wealth of product information. However, the high volume of reviews that are typically published for a single product makes harder for
individuals as well as manufacturers to locate the best reviews and understand the true underlying quality of a product. In this paper, we re-examine
the impact of reviews on economic outcomes like product sales and see how different factors affect social outcomes such as their perceived
usefulness. Our approach explores multiple aspects of review text, such as subjectivity levels, various measures of readability and extent of spelling
errors to identify important text-based features. In addition, we also examine multiple reviewer-level features such as average usefulness of past
reviews and the self-disclosed identity measures of reviewers that are displayed next to a review. Our econometric analysis reveals that the extent of
subjectivity, informativeness, readability, and linguistic correctness in reviews matters in influencing sales and perceived usefulness. Reviews that
have a mixture of objective, and highly subjective sentences are negatively associated with product sales, compared to reviews that tend to include
only subjective or only objective information. However, such reviews are rated more informative (or helpful) by other users. By using Random Forest
based classifiers, we show that we can accurately predict the impact of reviews on sales and their perceived usefulness. We examine the relative
importance of the three broad feature categories: ‘reviewer-related’ features, ‘review subjectivity’ features, and ‘review readability’ features, and find
that using any of the three feature sets results in a statistically equivalent performance as in the case of using all available features. This paper is the
first study that integrates econometric, text mining, and predictive modeling techniques toward a more complete analysis of the information captured
by user-generated online reviews in order to estimate their helpfulness and economic impact.
!
1INTRODUCTION
With the rapid growth of the Internet, product related word-
of-mouth conversations have migrated to online markets,
creating active electronic communities that provide a wealth of
information. Reviewers contribute time and energy to generate
reviews, enabling a social structure that provides benefits both
for the users and the firms that host electronic markets. In
such a context, “who” says “what” and “how” they say it,
matters.
On the flip side, a large number of reviews for a single
product may also make it harder for individuals to track the
gist of users’ discussions and evaluate the true underlying
quality of a product. Recent work has shown that the dis-
tribution of an overwhelming majority of reviews posted in
online markets is bimodal [1]. Reviews are either allotted an
extremely high rating or an extremely low rating. In such
situations, the average numerical star rating assigned to a
product may not convey a lot of information to a prospective
buyer or to the manufacturer who tries to understand what
aspects of its product are important. Instead, the reader has to
read the actual reviews to examine which of the positive and
which of the negative attributes of a product are of interest.
So far, the best effort for ranking reviews for consumers
comes in the form of “peer reviewing” in review forums, where
customers give “helpful” votes to other reviews in order to
signal their informativeness. Unfortunately, the helpful votes
•
Anindya Ghose and Panagiotis G. Ipeirotis are with the Department of Information,
Operations, and Management Sciences, Leonarn N. Stern School of Business,
New York University, New York, NY, 10012.
E-mail: {aghose,panos}@stern.nyu.edu.
Manuscript received August 31, 2008; revised on June 30, 2009; revised on Dec 31,
2009.
are not a useful feature for ranking recent reviews: the helpful
votes are accumulated over a long period of time, and hence
cannot be used for review placement in a short- or medium-
term time frame. Similarly, merchants need to know what
aspects of reviews are the most informative from consumers’
perspective. Such reviews are likely to be the most helpful for
merchants, as they contain valuable information about what
aspects of the product are driving the sales up or down.
In this paper, we propose techniques for predicting the
helpfulness and importance of a review so that we can have:
•
a consumer-oriented mechanism which can potentially
rank the reviews according to their expected helpfulness
(i.e., estimating the social impact), and
•
a manufacturer-oriented ranking mechanism, which can
potentially rank the reviews according to their expected
influence on sales (i.e., estimating the economic impact).
To understand better what are the factors that influence
consumers perception of usefulness and what factors affect
consumers most, we conduct a two-level study. First, we
perform an explanatory econometric analysis, trying to identify
what aspects of a review (and of a reviewer) are important
determinants of its usefulness and impact. Then, at the second
level we build a predictive model using Random Forests that
offer significant predictive power and allow us to predict with
high accuracy how peer consumers are going to rate a review
and how sales will be affected by the posted review.
Our algorithms are based on the idea that the writing
style of the review plays an important role in determining
the perceived helpfulness by other fellow customers and
the degree of influencing purchase decisions. In our work,
we perform multiple levels of automatic text analysis to
identify characteristics of the review that are important. We
SUBMITTED FOR PUBLICATION AT IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 2
perform our analysis at the lexical, grammatical, semantic, and
at the stylistic levels to identify text features that have high
predictive power in identifying the perceived helpfulness and
the economic impact of a review. Furthermore, we examine
whether the past history and characteristics of a reviewer can
be a useful predictor for the usefulness and impact of a review.
We present an extensive experimental analysis using a real
data set of 411 products, monitored over a 15-month period
on Amazon.com. Our analysis indicates that we can predict
accurately the helpfulness and influence of product reviews.
The rest of the paper is structured as follows. First, Section 2
discusses related work and provides the theoretical framework
for generating the variables for our analysis. Then, in Section 3,
we describe our data set and discuss how we extract the
variables that we use to predict the usefulness and impact of a
review. In Section 4, we present our explanatory econometric
analysis for estimating the influence of the different variables
and in Section 5 we describe the experimental results of
our predictive modeling that uses Random Forest classifiers.
Finally, Section 6 provides some additional discussion and
concludes the paper.
2THEORETICAL FRAMEWORK AND RELATED LIT-
ERATURE
From a business perspective, consumer product reviews are
most influential if they affect product sales and the online
behavior of users of the word-of-mouth forum.
2.1 Sales Impact
The first relevant stream of literature assesses the effect of
online product reviews on sales. Research in this direction
has generally assumed that the primary reason that reviews
influence sales is because they provide information about the
product or the vendor to potential consumers.
Prior research has demonstrated an association between
numeric ratings of reviews (review valence) and subsequent
sales of the book on that site [2], [3], [4] or between review
volume and sales [5], [6], [7]. Indeed, to the extent that
better products receive more positive reviews, there should
be a positive relationship between review valence and sales.
Research also demonstrated that reviews and sales may be
positively related even when underlying product quality is
controlled [3], [5].
However, prior work has not looked at how the textual
characteristics of a review affect sales. Our hypothesis is that
the text of product reviews affects sales even after taking
into consideration the numerical information such as review
valence and volume. Intuitively, reviews of reasonable length,
that are easy to read, and lack spelling and grammar errors
should be, all else being equal, more helpful and influential
compared to other reviews that are difficult to read and have
errors. Reviewers also write “subjective opinions” that portray
reviewers’ emotions about product features or more “objective
statements” that portray factual data about product features,
or a mix of both.
Keeping these in mind, we formulate three potential con-
structs for text-based features that are likely to have an impact:
(i) the average level of subjectivity and the range and mix of
subjective and objective comments, (ii) the extent to which
the content is easy to read, and (iii) the proportion of spelling
errors in the review. In particular, we test the following
hypotheses:
Hypothesis 1a
: All else equal, a change in the subjectivity level
and mixture of objective and subjective statements in reviews will
be associated with a change in sales.
Hypothesis 1b
: All else equal, a change in the readability score of
reviews will be associated with a change in sales.
Hypothesis 1c
: All else equal, a decrease in the proportion of
spelling errors in reviews will be positively related to sales.
2.2 Helpfulness Votes and Peer Recognition
A second stream of related research on word-of-mouth sug-
gests that perceived attributes of the reviewer may shape
consumer response to reviews [5]. In the social psychology
literature, message source characteristics have been found to
influence judgment and behavior [8], [9], [10], [11], and it has
been often suggested that source characteristics might shape
product attitudes and purchase propensity. Indeed, Forman et
al. [5] draw on the information processing literature to suggest
that product sales will be affected by reviewer disclosure of
identity-related information. Prior research on computer medi-
ated communication (CMC) suggests that online community
members communicate information about product evaluations
with an intent to influence others’ purchase decisions as well
as provide social information about contributing members
themselves [12], [13]. Research concerning the motivations
of content creators in online contexts highlights the role
of identity motives in defining why users provide social
information about themselves (e.g., [14], [15], [16], [17]).
Increasingly, we have seen that both identity-descriptive
information about reviewers and product information is
prominently displayed on the web sites of online retailers.
Prior research on self-verification in online contexts has
pointed out the use of persistent labeling, defined as using
a single, consistent way of identifying oneself such as ’real
name’ in the Amazon context, and self-presentation, defined
as ways of presenting oneself online that may help others
to identify one, such as posting geographic location or a
personal profile in the Amazon context [17] as important
phenomena in the online world. Indeed, information about
product reviewers is often highly salient. Visitors to the site
can see more professional aspects of reviewers such as their
badges (e.g., “top-50 reviewer,” “top-100 reviewer” badges)
and ranks (“reviewer rank”) as well as personal information
about reviewers ranging from their real name to where they
live, their nick names, hobbies, professional interests, pictures
and other posted links. In addition, users have the opportunity
to examine more “professional” aspects of a reviewer such
as the proportion of helpful votes given by other users not
only for a given review but across all the reviews of all other
products posted by a reviewer. Further, interested users can
also read the actual content of all reviews generated by a
reviewer across all products.
With regard to the benefits reviewers derive, work on online
user-generated content has primarily focused on the conse-
quences of peer recognition rather than on its antecedents [18],
[19]. Its only recently that [5] evaluated the influence of review-
ers’ disclosure of information about themselves on the extent
SUBMITTED FOR PUBLICATION AT IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 3
of peer recognition of reviewers and their interactions with the
review valence by drawing on the social psychology literature.
We hypothesize that after controlling for features examined in
prior work such as reviewer disclosure of identity information
and the valence of reviews, the actual text of the review matters
in determining the extent to which users find the review
useful. In particular, we focus on four constructs, namely
subjectiveness, informativeness, readability, and proportion
of spelling errors. Our paper thus contributes to the existing
stream of work by examining text-based antecedents of peer
recognition in online word-of-mouth forums. In particular, we
test the following hypotheses:
Hypothesis 2a
: All else equal, a change in the subjectivity level
and mixture of objective and subjective statements in a review will
be associated with a change in the perceived helpfulness of that
review.
Hypothesis 2b
: All else equal, a change in the readability of a
review will be associated with a change the perceived helpfulness of
that review.
Hypothesis 2c
: All else equal, a decrease in the proportion of
spelling errors in a review will be positively related to perceived
helpfulness of that review.
Hypothesis 2d
: All else equal, an increase in the average helpfulness
of a reviewer’s historical reviews will be positively related to perceived
helpfulness of a review posted by that reviewer.
This paper builds on our previous work [20], [21], [22].
In [20], [21] we examined just the effect of subjectivity, while
in the current work, we expanding our data to include more
product categories and examine a significantly increased
number of features, such as different readability metrics,
information about the reviewer history, different features of
reviewer disclosure and so on. The present paper is unique in
looking at how various additional features of the review text
affects product sales and the perceived helpfulness of these
reviews.
In parallel with our work, researchers in the natural lan-
guage processing field have examined the task of predicting
review helpfulness [23], [24], [25], [26], [27], using reviews
from Amazon.com or movie reviews as training and test data.
Our work uses a superset of the features used in the past for
helpfulness prediction (e.g., reviewer history and disclosure,
deviation of subjectivity in the review, and so on). Also, none
of these studies attempts to predict the influence of reviews
on product sales. A differentiating factor of our approach is
the two-pronged approach building on methodologies from
economics and from data mining, building both explanatory and
predictive models to understand better the impact of different
factors. Interestingly, all prior research use support vector
machines (in a binary classification and in regression mode),
which we observed to perform worse than Random Forests
(as we discuss in Section 5). Predicting the helpfulness of a
review is also related to the task of evaluating the quality of
web posts or the quality of answers to posted questions [28],
[29], [30], [31], although there are more cues (e.g., clickstream
data) that can be used to estimate the perceived quality of a
posting. Recently, Hao et al. [32] also presented techniques for
predicting whether a review will receive any votes about
its helpfulness or whether it will stay unrated. Tsur and
Rappoport [33] presented an unsupervised algorithm for
estimating ranking the reviews according to their expected
helpfulness.
3DATA SET AND VARIABLES
A major goal of this paper is to explore how the user-generated
textual content of a review and the self-reported characteristics
of the reviewer who generated the review can influence
economic transactions (such as product sales) and online
community and social behavior (such as peer recognition in
the form of helpful votes). To examine this, we collected data
about the economic transactions on Amazon.com and analyzed
the associated review system. In this section, we describe the
data that we collected from Amazon; furthermore, we discuss
how we computed the variables to perform our analysis, based
on the discussion of Section 2.
3.1 Product and Sales Data
To conduct our study, we created a panel data set of products
belonging to three product categories:
1) Audio and video players (144 products),
2) Digital cameras (109 products), and
3) DVDs (158 products).
We picked the products by selecting all the items that appeared
in the “Top-100” list of Amazon over a period of 3 months
from January 2005 to March 2005. We decided to use popular
products, in order to have products in our study with a
significant number of reviews. Then, using Amazon web
services, from March 2005 until May 2006 we collected the
information for these products described below.
We collected various product-specific characteristics over
time. Specifically, we collected the manufacturer suggested list
price of the product, its Amazon retail price, its Amazon sales
rank (which serves as a proxy for units of demand [34], as we
will describe later).
Together with sales and price data, we also collected other
data that may influence the purchasing behavior of consumers.
For example, we collected the date the product was released
into the market, to compute the elapsed time from the date of
product release, since products released long time ago tend to
see a decrease in sales over time. We also collected the number
of reviews and the average review rating of the product over
time.
3.2 Individual Review Data
Beyond the product-specific data, we also collected all reviews
of a product since the product was released into the market.
For each review, we retrieve the actual textual content of
the review and the review rating of the product given by the
reviewer. The rating that a reviewer allocates to the reviewed
product is denoted by a number of stars on a scale of 1 to 5.
From the textual content, we generated a set of variables at
the lexical, grammatical, and at the stylistic level. We describe
these variables in detail in Section 3.4, when we describe the
textual analysis that we conducted.
Review Helpfulness:
Amazon has a voting system whereby
community members provide helpful votes to rate the reviews
of other community members. Previous peer ratings appear
SUBMITTED FOR PUBLICATION AT IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 4
Type Variable Explanation
Product and Sales Data
Retail Price The retail price at Amazon.com
Sales Rank The sales rank within the product category
Average Rating Average rating of the posted reviews
Number of Reviews Number of reviews posted for the product
Elapsed Date Number of days since the release of the product
Individual Review
Moderate Review Does the Review have a moderate rating (3 star rating) or not
Helpful Votes The number of helpful votes for the review
Total Votes The total number of votes for the review
Helpfulness
HelpfulVotes
Tota lVot es
Reviewer Characteristics
Reviewer Rank The reviewer rank according to Amazon
Top-10 Reviewer Is the reviewer a Top-10 reviewer?
Top-50 Reviewer Is the reviewer a Top-50 reviewer?
Top-100 Reviewer Is the reviewer a Top-100 reviewer?
Top-500 Reviewer Is the reviewer a Top-500 reviewer?
Real Name Has the reviewer disclosed his/her real name?
Nickname Does the reviewer have a nickname listed in the profile?
Hobbies Does the reviewer have an ”about me” section in the profile?
Birthday Does the reviewer list his/her birthday?
Location Does the reviewer disclose its location?
Web Page Does the reviewer have a home page listed?
Interests Does the reviewer list his/her interests?
Snippet Does the reviewer has a description in the reviewer profile?
Any Disclosure Does the reviewer list any of the above in the reviewer profile?
Reviewer History
Number of Past Reviews Number of reviews posted by the reviewer
Reviewer History Macro Average past review helpfulness (macro-averaged)
Reviewer History Micro Average past review helpfulness (micro-averaged)
Past Helpful Votes Number of helpful votes accumulated in the past from the reviewer
Past Total Votes Number of total votes on the reviews posted in the past for the reviewer
Review Readability
Length (Chars) The length of the review in characters
Length (Words) The length of the review in words
Length (Sentences) The length of the review in sentences
Spelling Errors The number of spelling errors in the review
ARI The Automated Readability Index (ARI) for the review
Gunning Index The Gunning–Fog index for the review
Coleman–Liau index The Coleman–Liau index for the review
Flesch Reading Ease The Flesch Reading Ease score for the review
Flesch–Kincaid Grade Level The Flesch–Kincaid Grade Level for the review
SMOG The Simple Measure of Gobbledygook score for the review
Review Subjectivity
AvgProb The average probability of a sentence in the review being subjective
DevProb The standard deviation of the subjectivity probability
TABLE 1
The variables collected for our study. The panel data set contains data collected over a period of 15 months; we collected the
variables daily and we capture the variability over time for the variables that change over time (e.g., sales rank, price, reviewer
characteristics and so on).
immediately above the posted review, in the form, “[number
of helpful votes] out of [number of members who voted] found the
following review helpful.” These helpful and total votes enable
us to compute the fraction of votes that evaluated the review
as helpful. To have as much accurate representation of the
percentage of customers that found the review helpful, we
collected the votes in December 2007, ensuring that there is a
significant time period after the time the review was posted
and that there is a significant number of peer rating votes
accumulated for the review.
3.3 Reviewer Characteristics
Reviewer Disclosure:
While review valence is likely to
influence consumers, there is reason to believe that social
information about reviewers themselves (rather than the
product or vendor) is likely to be an important predictor
of consumers’ buying decisions [5]. On many sites, social
information about the reviewer is at least as prominent as
product information. For example, on sites such as Amazon,
information about product reviewers is graphically depicted,
highly salient, and sometimes more detailed and voluminous
than information on the products they review: the “Top-1000”
reviewers have special tags displayed next to their names, the
reviewers that disclose their real name
1
are also highlighted
and so on. Given the extent and salience of available social
information regarding product reviewers, it seems important
to control for the impact of such information on online product
sales and review helpfulness. Amazon has a procedure by
which reviewers can disclose personal information about
themselves. There are several types of information that users
can disclose: we focus our analysis on the categories most
commonly indicated by users: whether the user disclosed
their real name, their location, nickname, and hobbies. With
real name, we refer to a registration procedure that Amazon
provides for users to indicate their actual name by providing
verification with a credit card, as mentioned above. Reviewers
1.
Amazon compares the name of the reviewer with the name listed in the
credit card on file before assigning the “Real Name” tag.
SUBMITTED FOR PUBLICATION AT IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 5
may also post additional information in their profiles such
as geographical location, disclose additional information (e.g.,
“Hobbies”) or use a nickname (e.g., “Gadget King”). We use
these data to control for the impact of self-descriptive identity
claims. We encode this information as binary variables. We
also constructed an additional dummy variable, labeled “any
disclosure”; this variable captures each instance where the
reviewer has engaged in any one of the four kinds of self-
disclosure. We also collected the reviewer rank of the reviewer
as published on Amazon.
Reviewer History:
Since one of our goal is to predict the
future usefulness of a review, we wanted to examine whether
the past history of a reviewer can be used to predict the
usefulness of the future reviews written by the same reviewer.
For this, we collected the past reviews for each reviewer,
and collected the helpful and total votes for each of the
past reviews. Using this information, we constructed for each
reviewer and for each point in time the past performance of
a reviewer. Specifically, we created two variables, by micro-
averaging and macro-averaging the past votes on the reviews.
The variable reviewer history macro, is the ratio of all past
helpful votes divided by the total number of votes. Similarly,
we also created the variable reviewer history micro, in which
we first computed the average helpfulness for each of the
past reviews and then computed the average across all past
reviews. The difference with the macro and micro versions is
that the micro version gives equal weight to the helpfulness of
all past reviews, while the macro version weights more heavily
the importance of reviews that received a large number of
votes.
3.4 Textual Analysis of Reviews
Our approach is based on the hypothesis that the actual text of
the review matters. Previous text mining approaches focused
on extracting automatically the polarity of the review [35], [36],
[37], [38], [39], [40], [41], [42], [43], [44], [45], [46]. In our setting,
the numerical rating score already gives the (approximate)
polarity of the review,
2
so we look in the text to extract features
that are not possible to observe using simple numeric ratings.
Readability Analysis:
We are interested to examine what
types of reviews affect most sales and what types of reviews
are most helpful to the users. For example, everything else
being equal, a review that is easy to read will be more helpful
than another that has spelling mistakes and is difficult to read.
As a first, low-level variable, we measured the number
of spelling mistakes within each review, and we normalized
the number by dividing with the length of the review (in
characters).
3
To measure the spelling errors, we used an off-
the-shelf spell checker, ignoring capitalized words and words
with numbers in them. We also ignored the top-100 most
frequent non-English words that appear in the reviews: most
of them were brand names or terminology words that do not
appear in the spellcheckers list. Furthermore, to measure the
cognitive effort that a user needs in order to read a review,
we measured the length of a review in sentences, words, and
characters.
2.
We should note, though, that the numeric rating does not capture all the
polarity information that appears in the review [19].
3.
To take the logarithm of the normalized variable for errorless reviews,
we added one to the number of spelling errors before normalizing.
Beyond these basic features, we also used the extensive
results from research on readability. Past research has shown
that easy-reading text improves comprehension, retention, and
reading speed, and that the average reading level of the US
adult population is at the eighth grade level [47]. Therefore, a
review that can be read easily by a large number of users is also
expected to be rated by more users. Today there are numerous
metrics for measuring the readability of a text, and while none
of them is perfect, the computed measures correlate well with
the actual difficulty of reading a text. To avoid idiosyncratic
errors peculiar to a specific readability metric, we computed
a set of metrics for each review. Specifically, we computed the
following:
• Automated Readability Index
• Coleman–Liau Index
• Flesch Reading Ease
• Flesch–Kincaid Grade Level
• Gunning fog index
• SMOG
(See [48] for detailed description on how to compute each of
these metrics.) Based on research in readability, these metrics
are useful metrics for measuring how easy is for a user to
read a review.
Subjectivity Analysis:
Beyond the lower level spelling and
readability analysis, we also expect that there are stylistic
choices that affect the perceived helpfulness of a review.
We observed empirically that there are two types of listed
information, from the stylistic point of view. There are reviews
that list “objective” information, listing the characteristics of
the product, and giving an alternate product description that
confirms (or rejects) the description given by the merchant.
The other types of reviews are the reviews with “subjective,”
sentimental information, in which the reviewers give a very
personal description of the product, and give information that
typically does not appear in the official description of the
product.
As a first step towards understanding the impact of the
style of the reviews on helpfulness and product sales, we
rely on existing literature of subjectivity estimation from
computational linguistics [41]. Specifically, Pang and Lee [41]
described a technique that identifies which sentences in a
text convey objective information, and which of them contain
subjective elements. Pang and Lee applied their techniques
in a data set with movie review data set, in which they
considered as objective information the movie plot, and as
subjective the information that appeared in the reviews. In our
scenario, we follow the same paradigm. In particular, objective
information is considered the information that also appears in the
product description, and subjective is everything else.
Using this definition, we then generated a training set with
two classes of documents:
•
A set of “objective” documents that contains the product
descriptions of each of the products in our data set.
• A set of “subjective” documents that contains randomly
retrieved reviews.
Since we deal with a rather diverse data set, we constructed
separate subjectivity classifiers for each of our product cate-
gories. We trained the classifier using a Dynamic Language
Model classifier with
n
-grams (
n =8
) from the LingPipe
