scispace - formally typeset
Search or ask a question
Book ChapterDOI

A language modeling approach for temporal information needs

28 Mar 2010-pp 13-25
TL;DR: This work addresses information needs that have a temporal dimension conveyed by a temporal expression in the user’s query by integrating temporal expressions into a language modeling approach, thus making them first-class citizens of the retrieval model and considering their inherent uncertainty.
Abstract: This work addresses information needs that have a temporal dimension conveyed by a temporal expression in the user’s query Temporal expressions such as “in the 1990s” are frequent, easily extractable, but not leveraged by existing retrieval models One challenge when dealing with them is their inherent uncertainty It is often unclear which exact time interval a temporal expression refers to We integrate temporal expressions into a language modeling approach, thus making them first-class citizens of the retrieval model and considering their inherent uncertainty Experiments on the New York Times Annotated Corpus using Amazon Mechanical Turk to collect queries and obtain relevance assessments demonstrate that our approach yields substantial improvements in retrieval effectiveness

Summary (5 min read)

1 Introduction

  • Many information needs have a temporal dimension as expressed by a temporal phrase contained in the user’s query.
  • Existing retrieval models, however, often do not provide satisfying results for such temporal information needs, as the following examples demonstrate: A sports journalist, interested in FIFA World Cup tournaments during the 1990s, issues the query fifa world cup 1990s.
  • Documents with details on specific crusades, for instance, the Fourth Crusade that begun in 1202 would often not be among the retrieved results, unless they explicitly mention the 13th Century.
  • Consider, as one such document collection, the archive of the New York Times that covers the years 1851–2009.
  • The key problem here is that the actual meaning of many temporal expressions is uncertain, or more specifically, it is not clear which exact time interval they actually 3 refer to.

Organization

  • Section 2 puts their work in context with existing related research.
  • Section 4 describes how temporal expressions can be integrated into a language modeling approach.
  • Conducted experiments and their results are described in Section 5.

Time-Aware Retrieval Models

  • Li and Croft [20] and Dakka et al. [13] both propose language models that take into account publication times of documents, in order to favor, for instance, more recent documents.
  • Del Corso et al. [12] address the problem of ranking news articles, taking into account publication times but also their interlinkage.
  • Thus, all of the approaches mentioned are based on the publication times of documents.
  • None of the approaches, though, considers temporal expressions contained in the documents’ contents.
  • Kalczynski et al. [17] study the human perception of temporal expressions and propose a retrieval model for business news archives that takes into account temporal expressions.

Extraction of Temporal Expressions

  • Koen and Bender [19] describe the Time Frames system that extracts temporal expressions and uses them to augment the user experience when reading news articles, for instance, by displaying a temporal context of concurrent events.
  • Several prototypes are available that make use of temporal expressions when searching the Web, most notably, Google’s Timeline View [2] and TimeSearch [5].
  • Details about their internals, though, have not been published.

Crowdsourcing for IR Evaluation

  • Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) are becoming a common tool for conducting experiments in information retrieval.
  • AMT, as the best-known platform, allows requesters to publish so-called Human Intelligence Tasks (HITs), i.e., tasks that are hard for a computer but relatively easy for a human (e.g., determining the correct orientation of a photo).
  • Apart from that, requesters can restrict the workers allowed to take up their HITs, for instance, based on the geographical location or depending on whether the worker passes a qualification test.
  • On successful completion of a HIT, workers are paid a small reward that is typically below $0.10.

Time Domain & Temporal Expression Model

  • The authors apply a discrete notion of time and assume the integers Z as their time domain T with timestamps t ∈ T denoting the number of time units (e.g., milliseconds or days) passed (to pass) since a reference time-point (e.g., the UNIX epoch).
  • In their representation tbl and tbu are respectively a lower bound and upper bound for the begin boundary of the time interval – marking the time interval’s earliest and latest possible begin time.
  • The authors consider these time intervals thus as their elementary units of meaning in this work.
  • Note that for notational convenience the authors use the format YYYY/MM/DD to represent chronons – their actual values are integers as described above.

Collection & Query Model

  • The authors distinguish two modes of how they derive such a query from the user’s input, which differ in how they treat temporal expressions extracted from the input.
  • In the inclusive mode, the parts of the user’s input that constitute a temporal expression are still included in the textual part of the query.
  • In the exclusive mode, these are no longer included in the textual part.

4 Language Models for

  • With their formal model and notation established, the authors now turn their attention to how temporal expressions can be integrated into a language modeling approach, and how they can leverage them to improve retrieval effectiveness for temporal information needs.
  • In the second step, a temporal expression is generated from the temporal expression T just drawn.
  • Like other language modeling approaches, their model is prone to the zero-probability problem – if one of the query temporal expressions has zero probability of being generated from the document, the probability of generating the query from this document is zero.
  • In other words, a query temporal expression is more likely to be generated from a temporal expression that closely matches it.

Uncertainty-Ignorant Language Model

  • The authors first approach, further referred to as LmT, ignores the uncertainty inherent to temporal expressions.
  • According to the following definition, a temporal expression T can only generate itself.
  • Proof of Theorem 4.1 Despite its simplicity the approach still profits from the extraction of temporal expressions.
  • To illustrate this, consider the two temporal expressions “in the 1980s” and “in the ’80s”.
  • Both share the same formal representation in their model, so that LmT can generate a query containing one of them from a document containing the other.

Uncertainty-Aware Language Model

  • As explained in the introduction, for many temporal expressions the exact time interval that they refer to is uncertain.
  • The authors second approach LmtU explicitly considers this uncertainty.
  • The approach thus assumes equal likelihood for each time interval [qb, qe] that Q can refer to.
  • Intuitively, each time interval that the user may have had in mind when uttering Q is assumed equally likely.
  • Theorem 4.2 LmtU meets the requirements of specificity, coverage, and maximality defined above.

Efficient Computation

  • For a temporal resolution with chronons corresponding to days the total number of time intervals that this temporal expression can refer to is 66, 795.
  • The authors can compute |Q| using their preceding arguments.
  • Thus, the authors have shown that the generative model underlying LmtU allows for efficient computation.
  • This can be implemented efficiently by keeping a small inverted index in main memory that keeps track of the documents that contain a specific temporal expression.

Methods under Comparison

  • The mixture parameters γ and λ control the Jelinek-Mercer smoothing used when generating the textual part and the temporal part of the query, respectively.
  • Further, notice that their baseline Lm, which is not aware of temporal expressions, always only considers q text as determined using the inclusive mode, i.e., containing all terms from the user’s input.

Implementation Details

  • The authors implemented all methods in Java 1.6 keeping data in an Oracle 11g database.
  • All experiments described below were run on a single SUN V40z 16 machine having four AMD Opteron CPUs, 16GB RAM, a large networkattached RAID-5 disk array, and running Microsoft Windows Server 2003.
  • When processing the two document collections, the authors did not remove stopwords nor apply lemmatization/stemming.
  • Temporal expressions were extracted using TARSQI [24].
  • It annotates a given input document using the TimeML [4] markup language.

Document Collections

  • Table 5.1 shows additional statistics about the two datasets.
  • From the figures the authors observe that the mean document length is similar for both datasets.
  • Documents from WIKI, on average, contain more than twice as many temporal expressions as documents from NYT.

Queries

  • Since the authors target a specific class of information needs, query workloads used in benchmarks like TREC [6] are unemployable in their setting.
  • In their first study, workers were provided with an entity related to one of the topics Sports, Culture, Technology, or World Affairs and asked to specify a temporal expression that fits the given entity.
  • Isaac Singer invented the sewing machine, then 18 patented the motor for a sewing machine later in that decade, also known as sewing machine [1850s].
  • The Bulls won the NBA Finals that year, also known as chicago bulls [1991].
  • Among the queries obtained from their user studies, the authors selected the 40 queries shown in Figure 5.3.

Relevance Assessments

  • Relevance assessments were also collected using AMT.
  • Each of these query-document pairs was assessed by five workers on AMT.
  • Workers could state whether they considered the document relevant or not relevant to the query.
  • Examples of provided explanations are: roentgen [1895]: Wilhelm Roentgen was alive in 1895 when the building in New York at 150 Nassau Street in downtown Manhattan, NYC was built, they do not ever intersect other than sharing the same timeline of existence for a short while.
  • The article does not have any information on Keith Harring, only Laura Harring.

Experimental Results

  • The authors measure the retrieval effectiveness of the methods under comparison using Precision at k (P@k) and nDCG at k (N@k) as two standard measures.
  • When computing P@k, the authors employ majority voting.
  • When computing N@k, the average relevance grade assigned by workers is determined interpreting relevant as grade 1 and not relevant as grade 0. 22.

Overall Retrieval Performance

  • Table 5.2 and Table 5.3 give retrieval-effectiveness figures computed using all queries and cut-off levels k = 5 and k = 10 on NYT and WIKI, respectively.
  • For each of the five methods under comparison, the tables show the best-performing and worst-performing configuration with their corresponding values for the mixture parameters γ and λ.
  • For LmtU-EX the worst configuration beats the best configuration of the baseline.
  • Further, the worst and best configuration of LmtU-EX are close to each other demonstrating the method’s robustness.

Retrieval Performance by Topic

  • For the best-performing configuration of each method (as given in Table 5.2 and Table 5.3), the authors compute retrieval-effectiveness measures at cut-off level k = 10 and group them by topic.
  • 23 For NYT the resulting figures are shown in Table 5.4 and support their above observations.
  • Thus, LmtU-EX consistently achieves the highest retrieval effectiveness across all topics.
  • Further, the authors observe that all methods perform worst on queries from Technology.
  • The best performance varies per method and measure.

Retrieval Performance by Temporal Granularity

  • Table 5.6 gives the resulting figures for NYT.
  • Apart from that, the authors observe significant variations in retrieval effectiveness across temporal granularities for the baseline Lm.
  • The worst performance varies per method and measure.
  • 24 For WIKI the resulting figures given in Table 5.7 show a less distinct picture.
  • Thus, for queries containing a month or a year, the baseline Lm achieves the best retrieval effectiveness, although LmtU-EX is close behind.

Summary

  • The authors experimental evaluation leads us to the following findings.
  • When assessed on the whole of queries, LmtU consistently achieves superior retrieval performance on both datasets.
  • The uncertainty-ignorant LmT model, in contrast, deteriorates retrieval performance in comparison to the baseline.
  • For both methods, the exclusive mode of deriving the query from the user’s input performs better than its inclusive counterpart.
  • In summary, (i) considering the uncertainty inherent to temporal expressions is essential and (ii) excluding terms that constitute a temporal expression from the textual part of the query is beneficial.

6 Discussion & Outlook

  • The authors have developed a novel approach that integrates temporal expressions into a language model retrieval framework, taking into account the uncertainty inherent to temporal expressions.
  • Comprehensive experiments on the New York Times Annotated Corpus and a snapshot of the English Wikipedia, as two publicly-available large-scale document collections, with relevance assessments obtained using Amazon Mechanical Turk showed that their approach substantially improves retrieval effectiveness for temporal information needs.

Outlook

  • The authors focus in this work has been on temporal information needs disclosed by an explicit temporal expression in the user’s query.
  • Consider a query such as bill clinton arkansas that is likely to allude to Bill Clinton’s time as Governor of Arkansas between 1971 and 1981.
  • Even for queries that do not have a temporal intent behind them, the user profits from a result documents that contain diverse temporal expressions.
  • Thus, for a query such as vincent van gogh, a set of result documents discussing different periods in the famous painter’s life is preferable to a set of documents focused on his final years.

Did you find this useful? Give us your feedback

Content maybe subject to copyright    Report

A
Language Modeling Approach
for
Temporal Information Needs
Klaus Berberich
Srikanta Bedathur
Omar Alonso
Gerhard Weikum
MPI–I–2010–5–001 January 2010

Authors’ Addresses
Klaus Berberich
Max-Planck-Institut f¨ur Informatik
Campus E1 4
66123 Saarbr¨ucken
Germany
Srikanta Bedathur
Max-Planck-Institut f¨ur Informatik
Campus E1 4
66123 Saarbr¨ucken
Germany
Omar Alonso
Max-Planck-Institut f¨ur Informatik
Campus E1 4
66123 Saarbr¨ucken
Germany
Gerhard Weikum
Max-Planck-Institut f¨ur Informatik
Campus E1 4
66123 Saarbr¨ucken
Germany

Abstract
This work addresses information needs that have a temporal dimension con-
veyed by a temporal expression in the user’s query. Temporal expressions
such as “in the 1990s” are frequent, easily extractable, but not leveraged by
existing retrieval models. One challenge when dealing with them is their in-
herent uncertainty. It is often unclear which exact time interval a temporal
expression refers to.
We integrate temporal expressions into a language modeling approach,
thus making them first-class citizens of the retrieval model and consider-
ing their inherent uncertainty. Experiments on the New York Times Anno-
tated Corpus using Amazon Mechanical Turk to collect queries and obtain
relevance assessments demonstrate that our approach yields substantial im-
provements in retrieval effectiveness.
Keywords
Temporal Expressions, Language Models, Crowdsourcing

Contents
1 Introduction 2
2 Related Work 5
3 Model 7
4 Language Models for Temporal Information Needs 9
5 Experimental Evaluation 16
6 Discussion & Outlook 26
1

1 Introduction
Many information needs have a temporal dimension as expressed by a tempo-
ral phrase contained in the user’s query. Existing retrieval models, however,
often do not provide satisfying results for such temporal information needs,
as the following examples demonstrate:
A sports journalist, interested in FIFA World C up tournaments dur-
ing the 1990s, issues the query fifa world cup 1990s. Documents such
as the New York Times articles shown in Figure 1.1 would often not
be found by e xisting retrieval models, despite their obvious relevance
to the journalist’s information need. Similarly, a document stating
France won the FIFA World Cup in 1998 or a document published
in 1998 mentioning FIFA World Cup final in July would be missed.
This is because existing retrieval models are not aware of the semantic
connections between the temporal expressions “in 1998” and “in July”
contained in the documents and the user’s query temporal expression
“1990s”.
A historian, doing research on Christianization, issues the query 13th
century crusades. Documents with details on specific crusades, for in-
stance, the Fourth Crusade that begun in 1202 would often not be
among the retrieved results, unless they explicitly mention the 13th
Century. Again, the reason is that existing retrieval models lack the
knowledge about the semantic connections between temporal expres-
sions like “from 1202 until 1204” and “in 1202” contained in documents
and temporal expression “13th century” contained in the query.
Improving retrieval effectiveness for such temporal information needs is
an important objective for several reasons. First, a significant percentage of
queries has temporal information needs behind them about 1.5% of web
queries were found to contain an explicit temporal expression (as reported
in [22]) and about 7% of web queries have an implicit temporal intent (as
2

Citations
More filters
Journal ArticleDOI
TL;DR: YAGO2 as mentioned in this paper is an extension of the YAGO knowledge base, in which entities, facts, and events are anchored in both time and space, and it contains 447 million facts about 9.8 million entities.

1,186 citations

Journal ArticleDOI
30 Nov 2008
TL;DR: A new approach to evaluation called TERC is described, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.
Abstract: Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

362 citations

Journal ArticleDOI
TL;DR: A survey of the existing literature on temporal information retrieval is presented, categorize the relevant research, describe the main contributions, and compare different approaches to provide a coherent view of the field.
Abstract: Temporal information retrieval has been a topic of great interest in recent years Its purpose is to improve the effectiveness of information retrieval methods by exploiting temporal information in documents and queries In this article, we present a survey of the existing literature on temporal information retrieval In addition to giving an overview of the field, we categorize the relevant research, describe the main contributions, and compare different approaches We organize existing research to provide a coherent view, discuss several open issues, and point out some possible future research directions in this area Despite significant advances, the area lacks a systematic arrangement of prior efforts and an overview of state-of-the-art approaches Moreover, an effective end-to-end temporal retrieval system that exploits temporal information to improve the quality of the presented results remains undeveloped

212 citations

Proceedings ArticleDOI
06 Jun 2010
TL;DR: This tutorial discusses state-of-the-art methods, research opportunities, and open challenges along this avenue of knowledge harvesting, to automatically construct and maintain a comprehensive knowledge base of facts about named entities, their semantic classes, and their mutual relations as well as temporal contexts, with high precision and high recall.
Abstract: There are major trends to advance the functionality of search engines to a more expressive semantic level. This is enabled by the advent of knowledge-sharing communities such as Wikipedia and the progress in automatically extracting entities and relationships from semistructured as well as natural-language Web sources. Recent endeavors of this kind include DBpedia, EntityCube, KnowItAll, ReadTheWeb, and our own YAGO-NAGA project (and others). The goal is to automatically construct and maintain a comprehensive knowledge base of facts about named entities, their semantic classes, and their mutual relations as well as temporal contexts, with high precision and high recall. This tutorial discusses state-of-the-art methods, research opportunities, and open challenges along this avenue of knowledge harvesting.

161 citations

01 Jan 2011
TL;DR: The current research trends are reviewed and a number of interesting applications along with open problems are presented along with future work for this exciting field of information management.
Abstract: Time is an important dimension of any information space. It can be very useful for a wide range of information retrieval tasks such as document exploration, similarity search, summarization, and clustering. Traditionally, information retrieval applications do not take full advantage of all the temporal information embedded in documents to provide alternative search features and user experience. However, in the last few years there has been exciting work on analyzing and exploiting temporal information for the presentation, organization, and in particular the exploration of search results. In this paper, we review the current research trends and present a number of interesting applications along with open problems. The goal is to discuss interesting areas and future work for this exciting field of information management.

154 citations


Cites background from "A language modeling approach for te..."

  • ...describe a model for temporal information needs [7]....

    [...]

References
More filters
Book
01 Jan 2008
TL;DR: In this article, the authors present an up-to-date treatment of all aspects of the design and implementation of systems for gathering, indexing, and searching documents; methods for evaluating systems; and an introduction to the use of machine learning methods on text collections.
Abstract: Class-tested and coherent, this groundbreaking new textbook teaches web-era information retrieval, including web search and the related areas of text classification and text clustering from basic concepts. Written from a computer science perspective by three leading experts in the field, it gives an up-to-date treatment of all aspects of the design and implementation of systems for gathering, indexing, and searching documents; methods for evaluating systems; and an introduction to the use of machine learning methods on text collections. All the important ideas are explained using examples and figures, making it perfect for introductory courses in information retrieval for advanced undergraduates and graduate students in computer science. Based on feedback from extensive classroom experience, the book has been carefully structured in order to make teaching more natural and effective. Although originally designed as the primary text for a graduate or advanced undergraduate course in information retrieval, the book will also create a buzz for researchers and professionals alike.

11,804 citations

Journal ArticleDOI
01 Aug 1998
TL;DR: It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection and provide further proof of concept for the use of language models for retrieval tasks.
Abstract: In today's world, there is no shortage of information. However, for a specific information need, only a small subset of all of the available information will be useful. The field of information retrieval (IR) is the study of methods to provide users with that small subset of information relevant to their needs and to do so in a timely fashion. Information sources can take many forms, but this thesis will focus on text based information systems and investigate problems germane to the retrieval of written natural language documents. Central to these problems is the notion of "topic." In other words, what are documents about? However, topics depend on the semantics of documents and retrieval systems are not endowed with knowledge of the semantics of natural language. The approach taken in this thesis will be to make use of probabilistic language models to investigate text based information retrieval and related problems. One such problem is the prediction of topic shifts in text, the topic segmentation problem. It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection. Two complementary sets of features are studied individually and then combined into a single language model. The language modeling approach allows this problem to be approached in a principled way without complex semantic modeling. Next, the problem of document retrieval in response to a user query will be investigated. Models of document indexing and document retrieval have been extensively studied over the past three decades. The integration of these two classes of models has been the goal of several researchers but it is a very difficult problem. Much of the reason for this is that the indexing component requires inferences as to the semantics of documents. Instead, an approach to retrieval based on probabilistic language modeling will be presented. Models are estimated for each document individually. The approach to modeling is non-parametric and integrates the entire retrieval process into a single model. One advantage of this approach is that collection statistics, which are used heuristically for the assignment of concept probabilities in other probabilistic models, are used directly in the estimation of language model probabilities in this approach. The language modeling approach has been implemented and tested empirically and performs very well on standard test collections and query sets. In order to improve retrieval effectiveness, IR systems use additional techniques such as relevance feedback, unsupervised query expansion and structured queries. These and other techniques are discussed in terms of the language modeling approach and empirical results are given for several of the techniques developed. These results provide further proof of concept for the use of language models for retrieval tasks.

2,736 citations

Journal ArticleDOI
TL;DR: To define precisely what MTurk is and what it is not, it is hoped that this will point out opportunities for the community to deliberately value ethics above cost savings.
Abstract: Recently heard at a tutorial in our field: “It cost me less than one hundred bucks to annotate this using Amazon Mechanical Turk!” Assertions like this are increasingly common, but we believe they should not be stated so proudly; they ignore the ethical consequences of using MTurk (Amazon Mechanical Turk) as a source of labor. Manually annotating corpora or manually developing any other linguistic resource, such as a set of judgments about system outputs, represents such a high cost that many researchers are looking for alternative solutions to the standard approach. MTurk is becoming a popular one. However, as in any scientific endeavor involving humans, there is an unspoken ethical dimension involved in resource construction and system evaluation, and this is especially true of MTurk. We would like here to raise some questions about the use of MTurk. To do so, we will define precisely what MTurk is and what it is not, highlighting the issues raised by the system. We hope that this will point out opportunities for our community to deliberately value ethics above cost savings.

776 citations

Journal ArticleDOI
30 Nov 2008
TL;DR: A new approach to evaluation called TERC is described, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.
Abstract: Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.

362 citations

Frequently Asked Questions (12)
Q1. What are the contributions in this paper?

This work addresses information needs that have a temporal dimension conveyed by a temporal expression in the user ’ s query. The authors integrate temporal expressions into a language modeling approach, thus making them first-class citizens of the retrieval model and considering their inherent uncertainty. Experiments on the New York Times Annotated Corpus using Amazon Mechanical Turk to collect queries and obtain relevance assessments demonstrate that their approach yields substantial improvements in retrieval effectiveness. 

Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) are becoming a common tool for conducting experiments in information retrieval. 

In summary, (i) considering the uncertainty inherent to temporal expressions is essential and (ii) excluding terms that constitute a temporal expression from the textual part of the query is beneficial. 

a significant percentage of queries has temporal information needs behind them – about 1.5% of web queries were found to contain an explicit temporal expression (as reported in [22]) and about 7% of web queries have an implicit temporal intent (as2reported in [21]). 

as the best-known platform, allows requesters to publish so-called Human Intelligence Tasks (HITs), i.e., tasks that are hard for a computer but relatively easy for a human (e.g., determining the correct orientation of a photo). 

Like other language modeling approaches, their model is prone to the zero-probability problem – if one of the query temporal expressions has zero probability of being generated from the document, the probability of generating the query from this document is zero. 

The probability P ( Q | T ) according to Definition 4.7 can be computed efficiently without enumerating all time intervals that Q respectively T can refer to. 

Queries are categorized according to their topic and temporal granularity, giving us a total of 20 query categories, each of which contains two queries. 

The authors represent temporal expressions as quadruples to capture their inherent uncertainty – a formal representation that the authors adopt from Zhang et al. [25]. 

Koen and Bender [19] describe the Time Frames system that extracts temporal expressions and uses them to augment the user experience when reading news articles, for instance, by displaying a temporal context of concurrent events. 

Metzler et al. [21], most recently, identify so-called implicitly5temporal queries and propose a method to bias arbitrary ranking functions in favor of documents matching the user’s implicit temporal intent – this work, in contrast, proposes a self-contained language modeling approach that seamlessly integrates temporal expressions. 

In detail, the authors define the probability of generating Q from the document d asP ( Q | T ) = 1 |Q| ∑ [qb, qe]∈Q P ( [qb, qe] | T ) , (4.9)12where the sum ranges over all time intervals included in Q.