A language modeling approach for temporal information needs
Summary (5 min read)
1 Introduction
- Many information needs have a temporal dimension as expressed by a temporal phrase contained in the user’s query.
- Existing retrieval models, however, often do not provide satisfying results for such temporal information needs, as the following examples demonstrate: A sports journalist, interested in FIFA World Cup tournaments during the 1990s, issues the query fifa world cup 1990s.
- Documents with details on specific crusades, for instance, the Fourth Crusade that begun in 1202 would often not be among the retrieved results, unless they explicitly mention the 13th Century.
- Consider, as one such document collection, the archive of the New York Times that covers the years 1851–2009.
- The key problem here is that the actual meaning of many temporal expressions is uncertain, or more specifically, it is not clear which exact time interval they actually 3 refer to.
Organization
- Section 2 puts their work in context with existing related research.
- Section 4 describes how temporal expressions can be integrated into a language modeling approach.
- Conducted experiments and their results are described in Section 5.
Time-Aware Retrieval Models
- Li and Croft [20] and Dakka et al. [13] both propose language models that take into account publication times of documents, in order to favor, for instance, more recent documents.
- Del Corso et al. [12] address the problem of ranking news articles, taking into account publication times but also their interlinkage.
- Thus, all of the approaches mentioned are based on the publication times of documents.
- None of the approaches, though, considers temporal expressions contained in the documents’ contents.
- Kalczynski et al. [17] study the human perception of temporal expressions and propose a retrieval model for business news archives that takes into account temporal expressions.
Extraction of Temporal Expressions
- Koen and Bender [19] describe the Time Frames system that extracts temporal expressions and uses them to augment the user experience when reading news articles, for instance, by displaying a temporal context of concurrent events.
- Several prototypes are available that make use of temporal expressions when searching the Web, most notably, Google’s Timeline View [2] and TimeSearch [5].
- Details about their internals, though, have not been published.
Crowdsourcing for IR Evaluation
- Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) are becoming a common tool for conducting experiments in information retrieval.
- AMT, as the best-known platform, allows requesters to publish so-called Human Intelligence Tasks (HITs), i.e., tasks that are hard for a computer but relatively easy for a human (e.g., determining the correct orientation of a photo).
- Apart from that, requesters can restrict the workers allowed to take up their HITs, for instance, based on the geographical location or depending on whether the worker passes a qualification test.
- On successful completion of a HIT, workers are paid a small reward that is typically below $0.10.
Time Domain & Temporal Expression Model
- The authors apply a discrete notion of time and assume the integers Z as their time domain T with timestamps t ∈ T denoting the number of time units (e.g., milliseconds or days) passed (to pass) since a reference time-point (e.g., the UNIX epoch).
- In their representation tbl and tbu are respectively a lower bound and upper bound for the begin boundary of the time interval – marking the time interval’s earliest and latest possible begin time.
- The authors consider these time intervals thus as their elementary units of meaning in this work.
- Note that for notational convenience the authors use the format YYYY/MM/DD to represent chronons – their actual values are integers as described above.
Collection & Query Model
- The authors distinguish two modes of how they derive such a query from the user’s input, which differ in how they treat temporal expressions extracted from the input.
- In the inclusive mode, the parts of the user’s input that constitute a temporal expression are still included in the textual part of the query.
- In the exclusive mode, these are no longer included in the textual part.
4 Language Models for
- With their formal model and notation established, the authors now turn their attention to how temporal expressions can be integrated into a language modeling approach, and how they can leverage them to improve retrieval effectiveness for temporal information needs.
- In the second step, a temporal expression is generated from the temporal expression T just drawn.
- Like other language modeling approaches, their model is prone to the zero-probability problem – if one of the query temporal expressions has zero probability of being generated from the document, the probability of generating the query from this document is zero.
- In other words, a query temporal expression is more likely to be generated from a temporal expression that closely matches it.
Uncertainty-Ignorant Language Model
- The authors first approach, further referred to as LmT, ignores the uncertainty inherent to temporal expressions.
- According to the following definition, a temporal expression T can only generate itself.
- Proof of Theorem 4.1 Despite its simplicity the approach still profits from the extraction of temporal expressions.
- To illustrate this, consider the two temporal expressions “in the 1980s” and “in the ’80s”.
- Both share the same formal representation in their model, so that LmT can generate a query containing one of them from a document containing the other.
Uncertainty-Aware Language Model
- As explained in the introduction, for many temporal expressions the exact time interval that they refer to is uncertain.
- The authors second approach LmtU explicitly considers this uncertainty.
- The approach thus assumes equal likelihood for each time interval [qb, qe] that Q can refer to.
- Intuitively, each time interval that the user may have had in mind when uttering Q is assumed equally likely.
- Theorem 4.2 LmtU meets the requirements of specificity, coverage, and maximality defined above.
Efficient Computation
- For a temporal resolution with chronons corresponding to days the total number of time intervals that this temporal expression can refer to is 66, 795.
- The authors can compute |Q| using their preceding arguments.
- Thus, the authors have shown that the generative model underlying LmtU allows for efficient computation.
- This can be implemented efficiently by keeping a small inverted index in main memory that keeps track of the documents that contain a specific temporal expression.
Methods under Comparison
- The mixture parameters γ and λ control the Jelinek-Mercer smoothing used when generating the textual part and the temporal part of the query, respectively.
- Further, notice that their baseline Lm, which is not aware of temporal expressions, always only considers q text as determined using the inclusive mode, i.e., containing all terms from the user’s input.
Implementation Details
- The authors implemented all methods in Java 1.6 keeping data in an Oracle 11g database.
- All experiments described below were run on a single SUN V40z 16 machine having four AMD Opteron CPUs, 16GB RAM, a large networkattached RAID-5 disk array, and running Microsoft Windows Server 2003.
- When processing the two document collections, the authors did not remove stopwords nor apply lemmatization/stemming.
- Temporal expressions were extracted using TARSQI [24].
- It annotates a given input document using the TimeML [4] markup language.
Document Collections
- Table 5.1 shows additional statistics about the two datasets.
- From the figures the authors observe that the mean document length is similar for both datasets.
- Documents from WIKI, on average, contain more than twice as many temporal expressions as documents from NYT.
Queries
- Since the authors target a specific class of information needs, query workloads used in benchmarks like TREC [6] are unemployable in their setting.
- In their first study, workers were provided with an entity related to one of the topics Sports, Culture, Technology, or World Affairs and asked to specify a temporal expression that fits the given entity.
- Isaac Singer invented the sewing machine, then 18 patented the motor for a sewing machine later in that decade, also known as sewing machine [1850s].
- The Bulls won the NBA Finals that year, also known as chicago bulls [1991].
- Among the queries obtained from their user studies, the authors selected the 40 queries shown in Figure 5.3.
Relevance Assessments
- Relevance assessments were also collected using AMT.
- Each of these query-document pairs was assessed by five workers on AMT.
- Workers could state whether they considered the document relevant or not relevant to the query.
- Examples of provided explanations are: roentgen [1895]: Wilhelm Roentgen was alive in 1895 when the building in New York at 150 Nassau Street in downtown Manhattan, NYC was built, they do not ever intersect other than sharing the same timeline of existence for a short while.
- The article does not have any information on Keith Harring, only Laura Harring.
Experimental Results
- The authors measure the retrieval effectiveness of the methods under comparison using Precision at k (P@k) and nDCG at k (N@k) as two standard measures.
- When computing P@k, the authors employ majority voting.
- When computing N@k, the average relevance grade assigned by workers is determined interpreting relevant as grade 1 and not relevant as grade 0. 22.
Overall Retrieval Performance
- Table 5.2 and Table 5.3 give retrieval-effectiveness figures computed using all queries and cut-off levels k = 5 and k = 10 on NYT and WIKI, respectively.
- For each of the five methods under comparison, the tables show the best-performing and worst-performing configuration with their corresponding values for the mixture parameters γ and λ.
- For LmtU-EX the worst configuration beats the best configuration of the baseline.
- Further, the worst and best configuration of LmtU-EX are close to each other demonstrating the method’s robustness.
Retrieval Performance by Topic
- For the best-performing configuration of each method (as given in Table 5.2 and Table 5.3), the authors compute retrieval-effectiveness measures at cut-off level k = 10 and group them by topic.
- 23 For NYT the resulting figures are shown in Table 5.4 and support their above observations.
- Thus, LmtU-EX consistently achieves the highest retrieval effectiveness across all topics.
- Further, the authors observe that all methods perform worst on queries from Technology.
- The best performance varies per method and measure.
Retrieval Performance by Temporal Granularity
- Table 5.6 gives the resulting figures for NYT.
- Apart from that, the authors observe significant variations in retrieval effectiveness across temporal granularities for the baseline Lm.
- The worst performance varies per method and measure.
- 24 For WIKI the resulting figures given in Table 5.7 show a less distinct picture.
- Thus, for queries containing a month or a year, the baseline Lm achieves the best retrieval effectiveness, although LmtU-EX is close behind.
Summary
- The authors experimental evaluation leads us to the following findings.
- When assessed on the whole of queries, LmtU consistently achieves superior retrieval performance on both datasets.
- The uncertainty-ignorant LmT model, in contrast, deteriorates retrieval performance in comparison to the baseline.
- For both methods, the exclusive mode of deriving the query from the user’s input performs better than its inclusive counterpart.
- In summary, (i) considering the uncertainty inherent to temporal expressions is essential and (ii) excluding terms that constitute a temporal expression from the textual part of the query is beneficial.
6 Discussion & Outlook
- The authors have developed a novel approach that integrates temporal expressions into a language model retrieval framework, taking into account the uncertainty inherent to temporal expressions.
- Comprehensive experiments on the New York Times Annotated Corpus and a snapshot of the English Wikipedia, as two publicly-available large-scale document collections, with relevance assessments obtained using Amazon Mechanical Turk showed that their approach substantially improves retrieval effectiveness for temporal information needs.
Outlook
- The authors focus in this work has been on temporal information needs disclosed by an explicit temporal expression in the user’s query.
- Consider a query such as bill clinton arkansas that is likely to allude to Bill Clinton’s time as Governor of Arkansas between 1971 and 1981.
- Even for queries that do not have a temporal intent behind them, the user profits from a result documents that contain diverse temporal expressions.
- Thus, for a query such as vincent van gogh, a set of result documents discussing different periods in the famous painter’s life is preferable to a set of documents focused on his final years.
Did you find this useful? Give us your feedback
Citations
1,186 citations
362 citations
212 citations
161 citations
154 citations
Cites background from "A language modeling approach for te..."
...describe a model for temporal information needs [7]....
[...]
References
11,804 citations
7,318 citations
2,736 citations
776 citations
362 citations
Related Papers (5)
Frequently Asked Questions (12)
Q2. What are the common tools for conducting experiments in information retrieval?
Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) are becoming a common tool for conducting experiments in information retrieval.
Q3. What is the way to evaluate the retrieval effectiveness of a query?
In summary, (i) considering the uncertainty inherent to temporal expressions is essential and (ii) excluding terms that constitute a temporal expression from the textual part of the query is beneficial.
Q4. What is the reason for the lack of temporal information needs behind web queries?
a significant percentage of queries has temporal information needs behind them – about 1.5% of web queries were found to contain an explicit temporal expression (as reported in [22]) and about 7% of web queries have an implicit temporal intent (as2reported in [21]).
Q5. What is the best-known platform for releasing such tasks?
as the best-known platform, allows requesters to publish so-called Human Intelligence Tasks (HITs), i.e., tasks that are hard for a computer but relatively easy for a human (e.g., determining the correct orientation of a photo).
Q6. What is the probability of generating the temporal expression from a document?
Like other language modeling approaches, their model is prone to the zero-probability problem – if one of the query temporal expressions has zero probability of being generated from the document, the probability of generating the query from this document is zero.
Q7. How can the authors compute the probability P ( Q | T ) according to Definition 4.7?
The probability P ( Q | T ) according to Definition 4.7 can be computed efficiently without enumerating all time intervals that Q respectively T can refer to.
Q8. How many queries are in each category?
Queries are categorized according to their topic and temporal granularity, giving us a total of 20 query categories, each of which contains two queries.
Q9. What is the way to represent temporal expressions?
The authors represent temporal expressions as quadruples to capture their inherent uncertainty – a formal representation that the authors adopt from Zhang et al. [25].
Q10. What is the recent work that uses temporal expressions?
Koen and Bender [19] describe the Time Frames system that extracts temporal expressions and uses them to augment the user experience when reading news articles, for instance, by displaying a temporal context of concurrent events.
Q11. What is the recent work that considers temporal expressions?
Metzler et al. [21], most recently, identify so-called implicitly5temporal queries and propose a method to bias arbitrary ranking functions in favor of documents matching the user’s implicit temporal intent – this work, in contrast, proposes a self-contained language modeling approach that seamlessly integrates temporal expressions.
Q12. What is the probability of generating Q from the document d asP?
In detail, the authors define the probability of generating Q from the document d asP ( Q | T ) = 1 |Q| ∑ [qb, qe]∈Q P ( [qb, qe] | T ) , (4.9)12where the sum ranges over all time intervals included in Q.