Advertising Content and Consumer Engagement on Social Media: Evidence from Facebook

TL;DR: It is found that inclusion of widely used content related to brand personality is associated with higher levels of consumer engagement (Likes, comments, shares) with a message, and certain directly informative content, such as deals and promotions, drive consumers’ path to conversio...

Abstract: We describe the effects of social media advertising content on customer engagement using Facebook data. We content-code more than 100,000 messages across 800 companies using a combination of Amazon Mechanical Turk and state-of-the-art Natural Language Processing and machine learning algorithms. We use this large-scale dataset of content attributes to describe the association of various kinds of social media marketing content with user engagement - defined as Likes, comments, shares, and click-throughs - with the messages. We find that inclusion of widely used content related to brand-personality - like humor, emotion and brand’s philanthropic positioning - is associated with higher levels of consumer engagement (like, comment, share) with a message. We find that directly informative content - like mentions of prices and availability - is associated with lower levels of engagement when included in messages in isolation, but higher engagement levels when provided in combination with brand-personality content. We also find certain directly informative content such as the mention of deals and promotions drive consumers’ path-to-conversion (click-throughs). These results hold after correcting for the non-random targeting of Facebook’s EdgeRank (News Feed) algorithm, so reflect more closely user reaction to content, rather than Facebook’s behavioral targeting. Our results suggest therefore that there may be substantial gains from content engineering by combining informative characteristics associated with immediate leads (via improved click-throughs) with brand-personality related content that help maintain future reach and branding on the social media site (via improved engagement). These results inform content design strategies in social media. Separately, the methodology we apply to content-code large-scale textual data provides a framework for future studies on unstructured data such as advertising content or product reviews.

Summary

1 Introduction

• Models of informative advertising (c.f. Butters (1977); Grossman and Shapiro (1984)) allow for advertising to inform agents only about price and product existence − yet, casual observation and several studies in lab settings (c.f. Armstrong (2010)) suggest advertisements contain much more information and content beyond prices.
• While many brands have established a social media presence, it is not clear what kind of content works better and for which firm, and in what way.
• For each post, their data also contains time-series information on two kinds of engagement measures −.
• The authors main finding from the empirical analysis is that persuasive content drives social media engagement significantly.

2 Data

• The authors dataset is derived from the “pages” feature offered by Facebook.
• Pages enable companies to create profile pages and to post status updates, advertise new promotions, ask questions and push content directly to consumers.
• The authors data comprises posts served from firms’ pages onto the Facebook profiles of the users that are linked to the firm on the platform.
• Check out what the pros are wearing here: http://bit.ly/nyiPeW.”1.

2.1.1 Raw Data and Selection Criteria

• To collect the data, the authors partnered with an anonymous firm, henceforth referred to as Company X that provides analytical services to Facebook Page owners by leveraging data from Facebook’s Insights.
• The data also includes two consumer engagement metrics: the number of Likes and comments for each post each day.
• The authors leverage this information in the methodology they develop later for accounting for non-random assignment of posts to users by Facebook.
• The raw data contains about a million unique posts by about 2,600 unique companies.
• These finer categories are combined into 6 broader industry categories following Facebook’s page classification criteria.

2.1.2 Content-coded Data

• First, the authors contract with workers through AMT and tag 5,000 messages for a variety of content profiles.
• Best practices reported in the recent literature are used to ensure the quality of results from AMT and to improve the performance of the NLP algorithm (accuracy, recall, precision).
• The authors include these content categories to investigate more formally considerations laid out in industry white papers, trade-press articles and blog reports about the efficacy of message attributes in social media engagement.
• Table 3 shows sample messages taken from Walmart’s page in December 2012 and shows how the authors would have tagged them.
• The authors discuss their methods (which involve obtaining agreement across 9 tagging individuals) in section 2.2.

2.1.3 Data Descriptive Graphics

• This section presents descriptive statistics of the main stylized patterns in the data.
• Figure 2 shows box plots of the log of impressions, Likes, and comments versus the time (in days) since a post is released (τ).
• Emotional messages obtain the most number of Likes followed by posts identified as “likely to be posted by friends” (variable: FRIENDLIKELY).
• This means that 6 in 10 posts by celebrity pages in the data have some sort of small talk and/or content that does not relate to products or brands; and that there are no posts by celebrity owned pages that feature price comparisons.
• 10 11 Industry Category VS Message Content Appearance Percentage Biggest: Celebrity Smalltalk at 60.4% & Smallest: Celebrity PriceCompare at 0%.

2.2 Amazon Mechanical Turk

• The authors now describe their methodology for content-coding messages using AMT.
• Once a Turker tags more than 20 messages, a couple of tagged samples are randomly picked and manually examined for quality and performance.
• The authors believe their methodology for content-classification has good external validity.
• Finally, evaluating AMT based studies, Buhrmester et al. (2011) concludes that (1) Turkers are demographically more diverse than regular psychometric studies samples, and (2) the data obtained are at least as reliable as those obtained via traditional methods as measured by psychometric standards such as Cronbach’s Alpha, a commonly used inter-rater reliability measure.

2.3 Natural Language Processing (NLP) for Attribute Tagging

• Natural Language Processing is an interdisciplinary field composed of techniques and ideas from computer science, statistics and linguistics for enabling computers to parse, understand, store, and convey information in human language.
• When presented with a new set of sentences, the algorithm breaks these down to building blocks, identifies sentence-level attributes and assigns labels using the statistical models that were fine-tuned in the training process.
• The authors then utilize rule-based methods to identify brand and product mentions by looking up these lists.
• Finally, the authors utilize ensemble learning methods that combine classifications from the many classifiers and rule-based algorithms they use.
• This is repeated 10 times, each time using a different subset as the validation sample, and the performance measures averaged across the 10 runs.

3 Empirical Strategy

• Unfortunately, a complication arises because Facebook’s policy of delivery of messages to users is non-random: users more likely to find a post appealing are more likely to see the post in their newsfeed, a filtering implemented via Facebook’s “EdgeRank” algorithm.
• //whatisEdgeRank.com for a brief description of EdgeRank, also known as See http.
• Facebook categorizes post-type into 5 classes: status update, photo, video, app, or link.
• Time (τ) refers to the time since the post.
• The econometrics below sets up estimation using the aggregate post-level panel data split by demographics that the authors observe, while acknowledging the fact that non-random targeting is occurring at the individual-level.

3.1 First-stage: Approximating EdgeRank’s Assignment

• The authors represent post k’s type in a vector zk, the time since post k was released in τk, and the history of user i’s past engagement with company j on Facebook in a vector hijt.
• The authors will also estimate the right-hand function gd(.) separately for each demographic bucket, in effect allowing for slope heterogeneity in demographics in addition to intercept heterogeneity across demographics.
• S1 is a cubic spline smoothing function, essentially a piecewise-defined function consisting of many cubic polynomials joined together at regular intervals of the domain such that the fitted curve, the first and second derivatives are continuous.

3.2 Second-stage: Modeling Engagement given Post-Assignment

• The authors operationalize engagement via two actions, Likes and comments on the post.
• The selection functions Ψ̂(d)kjt serve as weights that reweigh the probability of Liking to account for the fact that those users were endogenously sampled, thereby correcting for the non-random nature of post assignment when estimating the outcome equation.
• 23 Maximizing the implied binomial likelihood across all the data, treating Ψ̂kjt as given, then delivers estimates of ψ.
• This essentially serves as a “quasi” control function that corrects for the selectivity in the second stage (Blundell and Powell, 2003), where the authors measure the effect of post characteristics on outcomes.
• The only post-characteristics used by EdgeRank for assignment is zk, which is controlled for.

4.1 First-Stage

• The first-stage model, as specified in Equation 3, approximates EdgeRank’s post assignment algorithm.
• For all demographics, the photo type has the highest coefficient (around 0.25) suggesting that photos are preferred to all other media types by EdgeRank.
• Figure 11 presents a box plot of the coefficients for τ across all 14 demographic bins.
• Finally, the coefficients for number of fans, N(d)jt , are positive and significant but they have relatively low magnitude.

4.2 Second-Stage

• In the second-stage, the authors measure the effect of content characteristics on engagement using their selectivitycorrected model from the first-stage.
• Interestingly, the interaction between persuasive and informative content is positive, implying that informative content increases engagement only in the presence of persuasive content in the message.
• This highlights the importance of EdgeRank correction.
• Looking at Likes, fewer persuasive content variables have positive impact but the results are qualitatively similar to that for comments.
• Similarly, the message type coefficients also vary by industry.

4.3 Out-of-Sample Prediction & Managerial Implications

• To conclude the paper, the authors assess the extent to which the models they develop may be used as an aid to content engineering, and to predict the expected levels of engagement for various hypothetical content profiles a firm may consider for a potential message it could serve to users.
• Then the authors discuss a back-of-the-envelope calculation to show how adding or removing particular content profiles may affect engagement outcomes for typical posts in their data.
• In the last two columns the authors present the predicted and actual ranks for the three messages in terms of their engagement.
• Now note that the standard deviation of the number of impressions is 129,874.
• For a message two standard deviations from the mean number of impressions, i.e., at 10,000 + 2×129,874 = 269,748 impressions, a 30% increase in comments and Likes translates to roughly an increase of 41 comments and 405 Likes, suggesting that content engineering can produce a fairly substantial increase in engagement for many posts.

5 Conclusions and Implications

• The authors show through a large-scale study that content engineering in social media has a significant impact on user engagement as measured by Likes and comments for posts.
• This presents a challenge to marketers who seek to build a large following on social media and who seek to leverage that following to disseminate information about new products and promotions.
• In addition, their results are moderated by industry type suggesting there is no one-size-fits-all content strategy and that firms need to test multiple content strategies.
• The authors find that posts mentioning holidays, especially by consumer product companies, have a negative effect on engagement.
• The authors hope this study contributes to improve content engineering by firms on social media sites and, more generally, creates interest in evaluating the effects of advertising content on consumer engagement.

Figures

Table 2: Variable Descriptions and Summary for Content-coded Data: To interpret the “Source” column, note that “Facebook” means the values are obtained from Facebook, “AMT” means the values are obtained from Amazon Mechanical Turk and “Computed” means it has been either calculated or identified using online database resources and rule-based methods in which specific phrases or content (e.g. brands) are matched. Finally, “AMT+Computed” means primary data has been obtained from Amazon Mechanical Turk and it has been further augmented with online resources and rule-based methods.

Table 14: Logistic Regression EdgeRank-Corrected Estimates Model Comparison (Likes)

Table 5: Performance of Text Mining Algorithm on 5000 Messages Using 10-fold Cross Validation: This table presents metrics for performance of the classification algorithms used. The left 3 columns show the metrics for the final algorithm which combines classifiers via ensemble learning method while the right 3 columns show the metric for a support vector machine algorithm. Notice that the support vector machine classifier tends to have low recall and high precision. Naive Bayes tends to have high recall but low precision. Classifiers on their own cannot successfully overcome precision-recall tradeoff (if one is higher, one is lower). But combining many different classifiers with ensemble learning can increase both precision and recall.

Figure 1: (Left) Example of a firm’s Facebook Page (Walmart). (Right) Example of a firm’s post and subsequent user engagement with that post (Tennis Warehouse). Example is not necessarily from our data.

Table 8: Persuasive vs Informative: Logistic regression for {Comment, Like} with composite summary variables for persuasive and informative content.

Figure 14: Survey Form Used in Amazon Mechanical Turk

Figure 9: Impression-Engagement Funnel: Facebook’s EdgeRank chooses subset of Page fans to show posts released by the page and fans who’ve seen the post engage with the post based on content and type. EdgeRank is modeled with generalized additive model and the final engagement is estimated through aggregate logistic regression. Details of estimation are in Sections 3.1 and 3.2.

Table 10: Predicted versus Actual Engagement Ranking for Three Illustrative Posts: Note: we anonymized some parts of the messages for presentation.

Figure 5: Average Likes and Comments by Message Content:This figure shows the average number of Likes and comments obtained by posts over their lifetime split by message content.

Figure 6: Bubble Chart of Broader Industry Category vs Message Content: This chart shows the relative percentage of message contents appearing within industry categories for 5,000 messages. Larger and lighter bubbles imply a higher percentage of messages in that cell. The largest bubble (60.4%) corresponds to SMALLTALK for the celebrity page category and the smallest bubble (0%) corresponds to PRICECOMPARE for the celebrity category.

Table 13: Logistic Regression EdgeRank-Corrected Estimates Model Comparison (Comments)

Figure 13: Logistic Regression by Industry (Comments and Likes): This bar graphs show the coefficients of logistic regression for EdgeRank-corrected model. Only the significant (at p<0.05 - but most are p<0.001) coefficients are graphed. In Like (right) graph, ASKCOMMENT for website is at -4.8 but zoomed in to make the graph look better.

Table 6: User-level Setup Notation

Figure 11: Time Since Post Release (τ) Coefficients Box plot Across Demographics: This box plot shows the coefficients on τ across all the demographics bin. τ = 1 is the base case and every coefficients are significant at the highest level of p < 0.001.

Figure 10: Page-level Fixed effect Estimates from Generalized Additive Model Across 14 Demographic Bins: This bar graph shows two randomly chosen page-level fixed effect estimates from the EdgeRank models. Only the statistically significant estimates are shown. New born clothing brands are positively significant for 18-24 female, 25-34 female, 35-44 female and 25-34 male. Protein bar brands have the highest fixed effect among 18-24 male demographics.

Figure 4: Average Likes and Comments by Message Type by Industry: This figure shows the average number of Likes and comments obtained by posts over their lifetime split by message type for each industry.

Figure 3: Average Likes and Comments by Message Type: This figure shows the average number of Likes and comments obtained by posts over their lifetime on Facebook, split by message type.

Table 11: Distribution of the Top (C1) and Top-two (C2) Concentration Ratios of the type of Posts Served by a Facebook Page over a Randomly picked 7-day period

Table 9: Aggregate Logistic Regression Results For Comments and Likes: This table presents the aggregate logistic regression on comments and Likes for both EdgeRank-corrected (ER) and uncorrected (NO ER) for all data. OR means Odds ratio and shows the odds ratio for the estimates left of the column.

Table 3: Examples of Messages and Their Content Tags: The messages are taken from 2012 December posts on Walmart’s Facebook page.

Figure 2: : Box Plots of Log(engagement+1) vs Time since Post Release: Three graphs show the box plots of (log) impressions, comments and Like vs. τ respectively. Both comments and Likes taper to zero after two and six days respectively. On the other hand, impressions die out slower. After 15 days, virtually all engagements and impressions (more than 99.9%) are accounted for. There are many outliers.

Full text

$3/:+67/8=4, +337=1:'3/'$3/:+67/8=4, +337=1:'3/'
").41'61=422437").41'61=422437
'60+8/3- '5+67 %.'6843')918=!+7+'6).

*:+68/7/3-438+38'3*43792+63-'-+2+3843"4)/'1+*/'*:+68/7/3-438+38'3*43792+63-'-+2+3843"4)/'1+*/'
:/*+3)+,642')+(440:/*+3)+,642')+(440
40=93++
'68/047'3'-'6
$3/:+67/8=4, +337=1:'3/'
'6/0+7.'/6
4114;8./7'3*'**/8/43'1;4607'8.88576+547/846=95+33+*92'60+8/3-&5'5+67
 '684,8.+*:+68/7/3-'3* 64248/43'3'-+2+3842243797/3+77*2/3/786'8/43
'3'-+2+38'3*5+6'8/43742243797/3+773'1=8/)742243797/3+77'3*46546'8+
42293/)'8/43742243742293/)'8/43#+).3414-='3*+;+*/'422437'60+8/3-
422437'7742293/)'8/43422437"4)/'1+*/'422437'3*8.+#+).3414-='3*334:'8/43
422437
!+)422+3*+*/8'8/43!+)422+3*+*/8'8/43
++47'3'-'6'/6*:+68/7/3-438+38'3*43792+63-'-+2+3843"4)/'1+*/'
:/*+3)+,642')+(440
'3'-+2+38")/+3)+
.885*<*4/46-237)
#./75'5+6/75478+*'8").41'61=422437.88576+547/846=95+33+*92'60+8/3-&5'5+67
46246+/3,462'8/4351+'7+)438')86+547/846=54(4<95+33+*9

*:+68/7/3-438+38'3*43792+63-'-+2+3843"4)/'1+*/':/*+3)+,642*:+68/7/3-438+38'3*43792+63-'-+2+3843"4)/'1+*/':/*+3)+,642
')+(440')+(440
(786')8(786')8
%+*+7)6/(+8.++,,+)84,74)/'12+*/''*:+68/7/3-)438+3843)97842+6+3-'-+2+3897/3-*'8',642
')+(440%+)438+38)4*+')+(4402+77'-+7')6477)425'3/+797/3-')42(/3'8/434,
2'>43+).'3/)'1#960'3*3'896'11'3-9'-+564)+77/3-'1-46/8.27%+97+8./7*'8'7+884789*=8.+
'774)/'8/434,:'6/4970/3*74,74)/'12+*/'2'60+8/3-)438+38;/8.97+6+3-'-+2+38A*+C3+*'7/0+7
)422+3877.'6+7'3*)1/)08.649-.7A;/8.8.+2+77'-+7%+C3*8.'8/3)197/434,;/*+1=97+*)438+38
6+1'8+*84(6'3*5+6743'1/8=A1/0+.9246'3*+248/43A/7'774)/'8+*;/8../-.+61+:+174,)43792+6
+3-'-+2+38/0+7)422+3877.'6+7;/8.'2+77'-+%+C3*8.'8*/6+)81=/3,462'8/:+)438+38A1/0+
2+38/4374,56/)+'3**+'17A/7'774)/'8+*;/8.14;+61+:+174,+3-'-+2+38;.+3/3)19*+*/32+77'-+7
/3/741'8/43(98./-.+6+3-'-+2+381+:+17;.+3564:/*+*/3)42(/3'8/43;/8.(6'3*5+6743'1/8=@6+1'8+*
'886/(98+7174)+68'/3*/6+)81=/3,462'8/:+)438+3879).'7*+'17'3*564248/437*6/:+)43792+67B5'8.
84)43:+67/43)1/)08.649-.7#.+7+6+791875+67/78',8+6/3)46546'8/3-)466+)8/437,468.+3436'3*42
8'6-+8/3-4,')+(440B7*-+!'30+;7++*'1-46/8.2'3*746+D+)8246+)147+1=97+66+')8/4384
)438+388.'3')+(440B7(+.':/46'18'6-+8/3-966+7918779--+788.'88.+6+'6+(+3+C8784)438+38
+3-/3++6/3-8.'8)42(/3+7/3,462'8/:+).'6')8+6/78/)78.'8.+15/34(8'/3/3-/22+*/'8+1+'*7:/'
/2564:+*)1/)08.649-.7;/8.(6'3*5+6743'1/8=@6+1'8+*)438+388.'8.+157/32'/38'/3/3-,9896+6+').
'3*(6'3*/3-438.+74)/'12+*/'7/8+:/'/2564:+*+3-'-+2+38#.+7+6+79187/3,462)438+38*+7/-3
786'8+-/+7"+5'6'8+1=8.+2+8.4*414-=;+'551=84)438+38)4*+8+<8/797+,91,46,9896+789*/+798/1/>/3-
937869)896+**'8'79).'7'*:+68/7/3-)438+3846564*9)86+:/+;7
+=;46*7+=;46*7
)43792+6+3-'-+2+3874)/'12+*/''*:+68/7/3-)438+38)438+38+3-/3++6/3-2'60+8/3-
)42293/)'8/432')./3+1+'63/3-3'896'11'3-9'-+564)+77/3-7+1+)8/43')+(440*-+!'30+;7
++*'1-46/8.2
/7)/51/3+7/7)/51/3+7
*:+68/7/3-'3* 64248/43'3'-+2+38?97/3+77?97/3+77*2/3/786'8/43'3'-+2+38'3*
5+6'8/437?97/3+773'1=8/)7?97/3+77'3*46546'8+42293/)'8/437?42293/)'8/43#+).3414-=
'3*+;+*/'?'60+8/3-?'7742293/)'8/43?"4)/'1+*/'?#+).3414-='3*334:'8/43
#./78+).3/)'16+5468/7':'/1'(1+'8").41'61=422437.88576+547/846=95+33+*92'60+8/3-&5'5+67

The Effect of Advertising Content on Consumer Engagement:
Evidence from Facebook
∗
Dokyun Lee
The Wharton School
Kartik Hosanagar
The Wharton School
Harikesh S. Nair
Stanford GSB
Abstract
We investigate the effect of social media content on customer engagement using a large-scale field
study on Facebook. We content-code more than 100,000 unique messages across 800 companies engaging
with users on Facebook using a combination of Amazon Mechanical Turk and state-of-the-art Natural
Language Processing algorithms. We use this large-scale database of advertising attributes to test the
effect of ad content on subsequent user engagement − defined as Likes and comments − with the mes-
sages. We develop methods to account for potential selectionbiasesthatarisefromFacebook’sfiltering
algorithm, EdgeRank, that assigns posts non-randomly to users. We find that inclusion of persuasive
content − like emotional and philanthropic content − increases engagemen t with a message. We find that
informative content − like mentions of prices, availability and product features − reduce engagement
when included in messages in isolation, but increase engagemen t when provided in combination with
persuasive attributes. Persuasive content thus seems to be the key to effective engagement. Our results
inform advertising design in social media, and the methodology we develop to content-code large-scale
textual data provides a framework for future studies on unstructured natural language data such as
advertising content or product reviews.
Keywords:advertising,socialmedia,advertisingcontent,large-scale data, natural language process-
ing, selection, Facebook, EdgeRank.
∗
We thank seminar participants at the ISIS Conference (2013),MackInstituteConference(Spring2013),andSCECR
Conference (Summer 2013) for comments, and a collaborating company that wishes to be anonymous for providing the data
used in the analysis. The authors gratefully acknowledge thefinancialsupportfromtheJayH.BakerRetailingCenterand
Mack Institute of the Wharton School and the Wharton Risk Center (Russell Ackoff Fellowship). All errors are our o wn.
1

1Introduction
Social media is increasingly taking up a greater share of consumers’ time spent online and, as a result, is
becoming a larger component of firm’s advertising budgets. Surveying 4,943 marketing decision makers at US
companies, the 2013 Chief Marketing Officer s ur vey (www.cmosurvey.org)reportsthatexpectedspending
on social media marketing will grow from 8.4% of firms’ total marketing budgets in 2013 to about 22% in
the next 5 years. As firms increase their social media activity, the role of content engineering has become
increasingly important. Content engineering seek s to develop ad content that better engage targeted users
and drive the desired goals of the marketer from the campaignstheyimplement. Surprisinglyhowever,
despite the numerous insights from the applied psychology literature about the design of the ad-creative
and its obvious relevance to practice, relatively little hasbeenformallyestablishedabouttheempirical
consequences of advertising content outside the labora tory, in real-world, field settings. Ad content also is
under emphasized in economic theory. The canonical economicmodelofadvertisingasasignal(c.f. Nelson
(1974); Kihlstrom and Riordan (1984); Milgrom and Roberts (1986)) does not postulate any direct role for ad
content because advertising intensity conveys all relevantinformationaboutproductqualityinequilibriumto
market participants. Models of informative advertising (c.f. Butters (1977); Grossman and Shapiro (1984))
allow for advertising to inform agents only about price and product existence − yet, casual observation and
several studies in lab settings (c.f. Armstrong (201 0 )) suggest advertisements contain much more information
and content beyond prices. In this paper, we investigate the role of content in driving consumer engagement
in social media in a field setting and document that content matters significantly. We find that a variety
of emotional, philanthropic and informative advertising content attributes affect engagement and that the
role of content varies significantly across firms and industries. The richness of our engagement data and the
ability to content code ads in a cost-efficient manner enables us to study the problem at a larger scale than
much of the previous literature on the topic.
Our analysis is of direct relevance to industry in better understanding and improving firms’ social media
marketing strategies. Recent studies (e.g., Creamer 2012) report that only about 1% of an average firm’s
Facebo ok fans (users who have Liked the Facebook Page of the firm) actually engage with the brand by
commenting on, Liking or sharing posts by the firm on the platform. As a result, designing better advertising
content that achieves superior reach and eng agement on s ocial media is an important issue for marketing on
this new medium. While many brands have established a social media presence, it is not clear what kind
of content works better and for which firm, and in what way. For example, are posts seeking to inform
consumers abo ut pr oduct or price attributes more effective than persuasive messages? Are videos or photos
more likely to engage users relative to simple status updates? Do messages explicitly soliciting user response
(e.g., “Like this post if ...”) draw more engagemen t or in fact turn users away? Does the same strategy apply
across different industries? Our pap er explores these kinds of questions and contributes to the formulation
of b etter content engineering policies in practice.
Our empirical investigation is implemented on Faceb ook, which is the largest social media platfor m in
the world. Many top brands now maintain a Facebook page from which they serve posts and messages to
connected users. This is a form of free social media advertising that has increasingly become a popular and
2

important channel for marketing. Our data comprises information on ab out 100,000 such messages posted
by a panel of about 800 firms over a 11-month period between Septem ber 2011 and July 2012. For each post,
our data also contains time-series information on two kinds of engagement measures − Likes and comments
− observed on Faceb ook. We supplement these engagement data with message attribute information that we
collect using a large-scale survey we implement on Amazon Mechanical T urk (henceforth “AMT”), combined
with a Natural Language Proces s ing algorithm (henceforth “NLP”) w e build to tag messages. We incorporate
new methods and procedures to improve the accuracy of contenttaggingonAMTandourNLPalgorithm.
As a result, our algorithm achieves about 99% accuracy, recall and precision for almost all tagged content
profiles. The methods we develop will be useful in future studies analyzing advertising content and product
reviews.
Our data also has several advantages that facilitate a study of advertising content. First, Faceb ook posts
have rich content attributes (unlike say, Twitter tweets, which are restricted in length) and rich data on
user engagement. Second, Facebook requires real names and, therefore, data on user activity on Facebook
is often more reliable compared to other social media sites. Third, engag e ment is meas ured on a daily basis
(panel data) by actual post-level engagement such as Likes and comments that are precisely tracked within
aclosedsystem.TheseaspectsmakeFacebookanalmostidealsetting to study the effect of ad content.
Our strategy for co ding content is motivated by the psychology, marketing and economic literatures
on advertising (see Cialdini (2001); Chandy et al. (2001); Bagwell (2007); Vakratsas and Ambler (1999)
for some representative overviews). In the economics literature, it is common to classify advertising as
informative (shifting beliefs about product existence or prices) or persuasive (shifting preferences directly).
The basis of informa tion is limited to prices and/or ex istence, and persuasive content is usually treated as
a“catch-all”withoutfinerclassification. Ratherthanthiscoarse distinction, our classification follows the
seminal classification work of Resnik and Stern (1977), who operationalize informative advertising based on
the number and characteristics of informational cues (see Abernethy and Franke, 1996 for an overview of
studies in this stream). Some criteria for classifying content as informative include details ab out product
deals, availability, price, and product related aspects that could be used in optimizing the purchase decision.
Following this stream, any product oriented facts, and brandandproductmentionsarecategorizedas
informative content. Following suggestions in the persuasion literature (Cialdini, 2001; Nan and Faber,
2004; Armstrong, 2010), we classify “persuasive” content asthosethatbroadlyseektoinfluencebyappealing
to ethos, pathos and logos strategies. For instance, the use of a celebrity to endorse a product or attempts to
gain trust or good-will (e.g., via small talk, banter) can be construed as the use o f ethos − appeals through
credibility o r character − and a form of persuasive advertising. Messages with philanthropic content that
induce empathy can be thought of as an attempt at persuasion via pathos − an app eal to a person’s emotions.
Lastly, messa ges with unusual or remarkable facts that influence consumers to adopt a product or capture
their attention can be categorized as persuasion via logos − an app eal through logic. We categorize content
that attempt to persuade and promote relationship building in this manner as persuasive content.
Estimation of the effect of content on subsequent engagement is complicated by the non-random allocation
of messages to users implemented by Facebook via its EdgeRank algorithm. EdgeRank tends to serve to
users posts that are newer and are expected to appeal better tohis/hertastes. Wedevelopcorrections
3

Frequently asked questions

Q1. What contributions have the authors mentioned in the paper "Advertising content and consumer engagement on social media: evidence from facebook" ?

The authors describe the effect of social media advertising content on customer engagement using data from Facebook. The authors use this data set to study the association of various kinds of social media marketing content with user engagement—defined as Likes, comments, shares, and click-throughs—with the messages. The authors find that directly informative content—like mentions of price and deals—is associated with lower levels of engagement when included in messages in isolation, but higher engagement levels when provided in combination with brand personality–related attributes. Separately, the methodology the authors apply to content-code text is useful for future studies utilizing unstructured data such as advertising content or product reviews. Their results suggest that there are benefits to content engineering that combines informative characteristics that help in obtaining immediate leads ( via improved click-throughs ) with brand personality–related content that helps in maintaining future reach and branding on the social media site ( via improved engagement ). 

Q2. What have the authors stated for future works in "Advertising content and consumer engagement on social media: evidence from facebook" ?

Here again, it is possible this effect may reduce in the future if firms start using emotional content excessively pushing consumer response to the region of declining returns. Future studies that evaluate other measures of interest can add value, particularly in validating the generalizability of their findings and in exploring mechanisms underpinning the effects the authors describe. There may be other measures worth considering, including whether users share posts with friends, visit the websites of firms posting messages, or buy more products from these firms. Kumar et al. ( 2013 ) show that social media can be used to generate growth in sales, and ROI, connecting social media metrics such as “ comments ” to financial metrics. 

Q3. What are some examples of sentence-level attributes and rules?

Some examples of sentence-level attributes and rules include: frequent noun words (bag-of-words approach), bigrams, the ratio of partof-speech used, tf-idf (term-frequency and inverse document frequency) weighted informative word weights, and whether “a specific key-word is present” rule. 

Q4. What are the practices used to ensure the quality of results from AMT?

Best practices reported in the recent literature are used to ensure the quality of results from AMT and to improve the performance of the NLP algorithm (accuracy, recall, precision). 

Q5. What is the role of content engineering in marketing?

Content engineering seeks to develop ad content that better engage targeted users and drive the desired goals of the marketer from the campaigns they implement. 

Q6. Why do the authors think their results have broad applicability?

Because of the scale of their study (over 800 firms and 100,000 messages analyzed), the authors believe their results generalize and have broad applicability. 

Q7. What is the effect of the interaction between persuasive and informative content?

the interaction between persuasive and informative content is positive, implying that informative content increases engagement only in the presence of persuasive content in the message. 

Q8. How many posts are removed after no significant activity?

Removing periods after which no significant activity is observed for a post reduces this to 665,916 rows of post-level snapshots (where activity is defined as either impressions, Likes, or comments). 

Q9. How do they show that social media can be used to generate growth in sales?

Kumar et al. (2013) show that social media can be used to generate growth in sales, and ROI, connecting social media metrics such as “comments” to financial metrics. 

Q10. What is the effect of the combining of results from a few Turkers?

Snow et al. (2008) show that combining results from a few Turkers can produce data equivalent in quality to that of expert labelers13for a variety of text tagging tasks. 

Q11. What is the complication of Facebook’s delivery of messages to users?

a complication arises because Facebook’s policy of delivery of messages to users is non-random: users more likely to find a post appealing are more likely to see the post in their newsfeed, a filtering implemented via Facebook’s “EdgeRank” algorithm. 

Q12. What age group are the impressions from the new-born clothing brand?

For posts by the the new-born clothing brand, the most impressions are among from females in the age-groups of 25-34, 18-24 and 35-44. 

Q13. Why is the coefficients for the number of fans positive?

This is because their model includes a smoothed term of the number of fans, s(N(d)jt ), which soaks up both the magnitude and nonlinearity. 

Q14. What is the method for combining the prediction from individual classifiers?

This step involves combining the prediction from individual classifiers by weightedmajority voting, unweighted-majority voting, or a more elaborate method called isotonic regression (Zadrozny and Elkan, 2002) and choosing the best performing method in terms of accuracy, precision and recall for each content profiles. 

Q15. How many coefficients are excluded from the table?

The authors exclude the 16 estimated τ coefficients from the table since they are all negative and statistically significant just as in the EdgeRank model in Figure 11. 

Q16. What are the intercepts for the post assignment algorithm?

The intercepts (θ(d)0 ) indicate that posts by companies in their dataset are shown most often to Females ages 35-44, Females 45-54 and Males 25-34. 

Q17. What is the effect of persuasive content on engagement?

The authors find that persuasive content has a positive and statistically significant effect on both types of engagement; further, informative content reduces engagement. 

Q18. What is the canonical economic model of advertising as a signal?

The canonical economic model of advertising as a signal (c.f. Nelson (1974); Kihlstrom and Riordan (1984); Milgrom and Roberts (1986)) does not postulate any direct role for ad content because advertising intensity conveys all relevant information about product quality in equilibrium to market participants. 

Q19. What is the likely explanation for the lack of engagement on Facebook?

One possible explanation is that near holidays, all Facebook pages indiscriminately mention holidays, leading to a dulled responses.