VOL. 21 NO. 4, DECEMBER, 2016
Contents | Author index | Subject index | Search |
Home
Moving beyond text highlights: inferring users' interests to
improve the relevance of retrieval
Vimala Balakrishnan, Yasir Mehmood and Yoganathan Nagappan
Abstract
Introduction. Studies have indicated that users' text highlighting
behaviour can be further manipulated to improve the relevance of
retrieved results. This article reports on a study that examined users'
text highlight frequency, length and users' copy-paste actions.
Method. A binary voting mechanism was employed to determine the
weights for the feedback, which were then used to re-rank the original
search results. A search engine prototype was built using the
Communications of the ACM test collection, with the well-known
BM25 acting as the baseline model.
Analysis. The proposed enhanced model's performance was
evaluated using the mean average precisions and F-score metrics,
and results were compared at the top 5, 10 and 15. Additionally,
comparisons were also made based on the number of terms used in a
query, that is single, double and triple terms.
Results. The findings show that the enhanced model significantly
outperformed BM25, and the rest of the models at all document levels.
To be specific, the enhanced model showed significant improvements
over the frequency model. Additionally, retrieval relevance was found
to be the best when the query length is two.
Conclusions. Users' post-click behaviour may serve as a significant
indicator of their interests, and thus can be used to improve the
relevance of the retrieved results. Future studies could look into
further extending this model by including other post-click behaviour
such as printing or saving.
Introduction
Online searching has become part of many people's work and
daily lives, including activities such as research, shopping and
entertainment (Clay and Esparza, 2012
). For example, it is
common for people to: seek information from Wikipedia, search
change font
through Google, buy products from eBay or Amazon.com, etc.
However, searching for relevant items or services can be a
daunting task due to the amount of information, and this is
further exacerbated by a lack of searching skills. Web users tend
to not know (or care) about the heterogeneity of Web content,
the syntax of query languages and the art of phrasing queries,
often resulting in them spending a lot of time looking for
relevant items on the Internet (Manning, Raghavan and Schutze,
2009; Varathan, Tengku Sembok, Abdul Kadir and Omar, 2014).
To solve the problem of users' lack of query skills, relevance
feedback is commonly used. Relevance feedback is a process
involving users in the development of information retrieval
systems, and aims to improve search results and increase user
satisfaction. According to Baeza-Yates and Ribeiro-Neto (1999
),
'in a relevance feedback cycle, the user is presented with a list of
the retrieved documents and, after examining them, marks
those which are relevant'. In fact, relevance feedback has been
shown to be an indicator of users' interests, which can then be
used to improve their satisfaction (Claypool, Le, Wased and
Brown, 2001; Fox, Karnawat, Mydland, Dumais and White,
2005; Liu, Gwizdka and Liu, 2010). Two components of
relevance feedback have evolved, namely, query expansion (i.e.,
automatic relevance feedback) and term reweighting. Query
expansion involves the addition of new terms to the initial query
automatically, using techniques such as pseudo-relevance
feedback (i.e., users get improved retrieval performance without
further interaction), thesaurus-based or other types of
expansions, with studies demonstrating the accuracy of the
interpretation of original queries to improve by this technique
(Belkin
et al., 2004; Crabtree, Andreae and Gao, 2007; Walker,
Robertson, Boughanem, Jones and Sparck, 1998). In contrast,
term reweighting refers to the modification of term weights
according to the relevance judgement by users (Baeza-Yates and
Ribeiro-Neto, 1999). In other words, this technique increases the
term weights in relevant documents whilst decreasing those in
irrelevant documents.
Relevance feedback is usually gathered through explicit and/or
implicit feedback. Explicit feedback requires the users to provide
feedback for products or services rendered, with methods
including ranking, rating, commenting and answering questions
(Balakrishnan, Ahmadi and Ravana, 2016
; Claypool et al., 2001;
Núñez-Valdéz
et al., 2012). Explicit feedback is well understood,
easy to implement and fairly precise. The approach, however,
requires the users to engage in additional activities beyond their
normal searching behaviour, hence resulting in higher user costs
in time and effort. Additionally, not all users like to be involved
in providing explicit feedback as the repetitive and frequent way
of obtaining the relevance judgement causes a cognitive overload
for the users (Claypool et al., 2001; Zemerli, 2012). Users are
generally busy reading or looking for the right document or item
during a search session, and thus they do not often provide
explicit feedback. In fact, during an experiment it was found that
only a small group of users agreed to give relevance judgments,
and sometimes they had to be paid to provide this information
(Spink, Jansen and Ozmultu, 2000
). Similarly, previous work on
GroupLens
found that users rated many fewer documents than
they read (Sarwar
et al., 1998). Thus, even though explicit
ratings are fairly precise in recognizing user interests, their
efficacy is limited.
On the other hand, implicit feedback such as mouse clicks (i.e.,
click-through data) can be mined unobtrusively and used to
determine users' preferences (Agichtein, Brill, Dumais and
Ragno, 2006; Agrawal, Halverson, Kenthapadi, Mishra and
Tsaparas, 2009; Carterette and Jones, 2008; Chuklin, Markov
and Rijke, 2015; Claypool et al., 2001; Dupret and Liao, 2010;
Feimin
et al., 2010; Joachims, Granka, Pan, Hembrooke and
Gay, 2005; Yu, Lu, Sun, and Zhang, 2012). For instance, mouse
clicks have been used for online advertising to estimate the
relevance of an advertisement to a search result page or a
document (Chatterjee, Hoffman and Novak, 2003
; Graepel,
Candela, Borchert and Herbrich, 2010).
One of the main difficulties in estimating relevance from click
data is due to position bias, that is, a document appearing in a
higher position is more likely to attract user clicks even though it
is irrelevant. Subsequent studies on implicit feedback hence
progressed to include users' post-click behaviour, which refers to
users' actions on the selected documents or results. These
include dwell time (i.e., reading or display time) (Balakrishnan
and Zhang, 2014; Fox et al., 2005; Oard and Kim, 1998),
printing (Oard and Kim, 1998
), scrolling (Claypool et al., 2001;
Guo and Agichtein, 2012
; Oard and Kim, 1998), and mouse or
cursor movements (Buscher, White, Dumais and Huang, 2012
;
Guo and Agichtein, 2010
; Huang, White and, Buscher, 2012),
with results indicating post-click behaviour to provide valuable
implicit feedback that could indicate the relevance of selected
documents. Huge volumes of implicit feedback data can be
gathered easily and unobtrusively. Furthermore, no mental
efforts are required from the users (Claypool
et al., 2001;
Manning
et al., 2009).
One of the recent implicit feedback techniques that has been
explored is text highlight or text selection, which involves
selecting a block of text to indicate its relevancy to the user.
Generally, people make some form of mark, such as highlights,
annotations, comments, circles, etc., on documents to indicate
interests or relevance (Shipman, Price, Marshall, Golovchinsky
and Schilit, 2003). Similar assumptions have been made in
information retrieval studies, whereby users' annotations and
text highlighting behaviour was used to improve document
relevance. However, research focusing on such behaviour is
scarce.
Studies of users' text highlighting behaviour thus far have
examined the frequency of text highlighting, that is, it is
assumed that the more text highlights a document contains, the
more relevant the document is to the user (Balakrishnan and
Zhang, 2014; White and Buscher, 2012). Determining a
document's relevance based only on the frequency of text
highlighting may be inadequate because factors such as the
length of the highlighted text and users' post-selection actions
may also indicate users' interests (White and Buscher, 2012
).
Furthermore, according to Buscher et al. (2012
), copy-paste and
reading aid were the two main reasons leading users to highlight
text. In fact, both text highlighting and copying are considered to
be very strong indicators of users' interests (Hauger, Paramythis,
and Weibelzahl, 2011; Hijikata, 2004). Therefore, these
indicators can potentially be used together to improve retrieval
relevance.
The current study aims to extend the above-mentioned works by
further exploring and manipulating users' text highlighting
behaviour. To be precise, the study intends to improve document
retrieval relevance by analysing three parameters: (i) frequency
of text highlight, (ii) length of text highlight, and (iii) user's copy-
paste action. The traditional ranking algorithm, Okapi BM25
was
used as the baseline (i.e., without users' feedback) and
Communications of the ACM (CACM) was used as the test
collection. To evaluate the retrieval effectiveness of the proposed
model, an experiment was conducted using a self-developed
prototype search engine. The retrieved results were analysed at
the top 5, 10 and 15 document levels, and also compared by
query lengths. As will be shown, the findings show that the
proposed model consistently yields significant improvements
over BM25, and the rest of the feedback models.
Related work
Implicit user feedback can be generally divided into two
categories: the query actions and the physical user reactions. The
query actions refer to ways in which the user interacts with the
search engine (e.g., clicks, key strokes) whereas the physical
reactions are users' unconscious behaviour (e.g., eye movements,
heart rate). Unlike the latter category, which requires special
devices to collect data, users' query actions can be easily
captured during a search session. The current study intends to
exploit these query actions, specifically users' text highlighting
behaviour.
Inferences drawn from implicit feedback are considered to be
less reliable compared to explicit feedback, but on the other
hand, large quantities of data can be gathered unobtrusively
(Jung, Herlocker, and Webster, 2007
). Studies focusing on
implicit feedback have investigated various user behaviour, such
as mouse clicks (Agichtein
et al., 2006; Agrawal et al., 2009;
Balakrishnan and Zhang, 2014
; Claypool et al., 2001; Dupret and
Liao, 2010; Feimin et al., 2010; Yu et al., 2012), dwell time
(Balakrishnan and Zhang, 2014
; Fox et al., 2005; Hassan, Jones
and Klinkner, 2010; Huang, White and Dumais, 2011), eye
tracking (Joachims
et al., 2007), and mouse movements
(Buscher
et al., 2012; Guo and Agichtein, 2010; Huang et al.,
2012), among others. Techniques such as mouse clicks are based
on the assumption that the clicked documents are relevant to the
search queries, however this may not be accurate. Joachims et
al. (2005
) reported two main issues: trust bias in which users
trusted the ranking quality of the search engine and only clicked
the first few results, and quality bias which refers to users'
varying behaviour for the same query in different search engines.
Generally, studies examine the search logs to understand users'
behaviour and interests because they can automatically capture
user interaction details. In addition, these data can be analysed
to optimize retrieval performances (Jordan, Simone, Thomas,
and Alexander, 2010), help query suggestions (Huanhuan et al.,
2008) and enhance the ranked results (Agichtein et al., 2006;
Balakrishnan and Zhang, 2014
). More recent studies have looked
into ways to improve retrieval by investigating other clicking
behaviour, such as Xu, Chen, Xu, Li, and Elbio (2010
) who used
click rate and last click to predict the relevant labels or Uniform
Resource Locators (URL). Although an overall improvement was
observed, using last click as an interest indicator may not be
accurate as well, because there are different reasons behind the
last click. For example, users who left the last documents may
have either succeeded in finding useful documents (good
abandon) or failed to find relevant documents (bad abandon),
and hence began a new search (Huang
et al., 2011).
Studies have also progressed into examining users' post-click
behaviour (i.e., actions performed after clicking on a link). A
simple technique would be the dwell time, whereby it is assumed
that if a document is relevant, the user may spend longer time on
it than other documents (Buscher, Elst, and Dengel, 2009
). In a
more recent study, dwell time was further experimented from
three different angles, that is, display time (i.e., interval time
between open and close of the document), dwell time (i.e.,
reading time) and decision time (i.e., decision-making time to
select document), with results indicating dwell time topped the
list in predicting document relevance compared to the other two