Rank aggregation methods for the Web
Summary (5 min read)
1. INTRODUCTION
- When there is just a single criterion (or \judge") for ranking, the task is relatively easy, and is simply a re ection of the judge's opinions and biases.
- (If simplicity w ere the only desideratum, dictatorship would prevail over democracy.).
- In contrast, this paper addresses the problem of computing a \consensus" ranking of the alternatives, given the individual ranking preferences of several judges.
- The authors call this the rank aggregation problem.
- The authors provide the theoretical underpinnings for stating criteria for \good" rank aggregation techniques and evaluating speci c proposals, and they o er novel algorithmic solutions.
1.1 Motivation
- As of February 2001, there were at least 24 general-purpose search engines (see Search Engine Watch 1]), as well as numerous special-purpose search engines.
- There are a number of good reasons why this is the case, even if the authors restrict attention to search engines that are meant to be \general purpose.".
- This is a problem conventionally studied in database middleware (see 15]).
- There is a second, very broad, set of scenarios where rank aggregation is called for.
- Notice that the former may produce no useful document, or too few of them, while the latter may produce an enormous list of documents where it is not clear which one to choose as the best.
1.2 Challenges
- The ideal scenario for rank aggregation is when each judge (search engine in the case of meta-search, individual criterion for multi-criteria selection, and subsets of queries in the case of word association queries) gives a complete ordering of all the alternatives in the universe of alternatives.
- This, however, is far too unrealistic for two main reasons.
- Secondly, search e ngines routinely limit access to about the rst few hundreds of pages in their rank-ordering.
- The issue of e ciency is also a serious bottleneck in performing rank aggregation for multi-criteria selection and word association queries.
- Therefore, any method for rank aggregation for Web ap-plications must be capable of dealing with the fact that only the top few hundred entries of each ranking are available.
1.3 Our results
- The authors provide a mathematical setting in which to study the rank aggregation problem, and propose several algorithms.
- By drawing on the literature from social choice theory, statistics, and combinatorial optimization, the authors formulate precisely what it means to compute a good consensus ordering of the alternatives, given several rankings of the alternatives.
- Besides the heuristics, the authors identify a crucial property of Kemeny optimal solutions that is particularly useful in combatting spam, and provide an e cient algorithm for minimally modifying any initial aggregation so as to enjoy this property.
- This property is called the \extended Condorcet criterion," and the authors call the e cient process that is guaranteed to achieve it \local Kemenization.".
- While there is no guarantee on the quality of the output, the latter methods are extremely e cient, and usually match or outperform the rst method.
1.4 Organization
- The authors describe their framework, including the notions of ranking, distance measures, and optimal aggregation in Section 2.
- This section also contains a brief description of concepts from graph theory and Markov c hains the authors need for this paper.
- Section 3 discusses spam, the extended Condorcet principle, and local Kemenization.
- Section 4 describes various rank aggregation methods, including the well-known Borda method and several other new methods.
2.1 Ranking
- (2) There are situations where full lists are not convenient or even possible.
- Let U denote the set of all Web pages in the world.
- Let denote the results of a search engine in response to some xed query.
- In other words, there are pages in the world which are unranked by this search engine with respect to the query.
- Such lists that rank only some of the elements in U are called partial lists. (3) A special case of partial lists is the following.
2.1.1 Distance measures
- After dividing this number by the maximum value jSj 2 =2, one can obtain a normalized value of the footrule distance, which is always between 0 and 1.
- The footrule distance between two lists can be computed in linear time.
- Dividing this numberby the maximum possible value ; jSj 2 the authors obtain a normalized version of the Kendall distance.
- Note that these distances are not necessarily metrics.
- The authors do not delve into such discussions here the interested reader can nd such arguments in the booksby Diaconis 12] , Critchlow 11] , or Marden 17] .
2.1.2 Optimal rank aggregation
- The aggregation obtained by optimizing Kendall distance is called Kemeny optimal aggregation and in a precise sense, corresponds to the geometric median of the inputs.
- The authors show that computing the Kemeny optimal aggregation is NP-Hard even when k = 4 .
- In Section 3 the authors establish a strong connection between satisfaction of the extended Condorcet criterion and ghting search engine \spam.".
- The following relation shows that Kendall distance can be approximated very well via the Spearman footrule distance.
- In Section 4 the authors exhibit a polynomial time algorithm to compute optimal footrule aggregation (scaled footrule aggregation for partial lists).
3. SPAM RESISTANCE AND CONDORCET CRITERIA
- This is called the extended Condorcet criterion (ECC).
- (If the evaluators are human, the typical scenario during the design and training of search engines, then the eventual product will incorporate the biases of the training evaluators.).
- In other words, under this de nition of spam, the spam pages are the Condorcet losers, and will occupy the bottom partition of any aggregated ranking that satis es the extended Condorcet criterion.
- This procedure is called local Kemenization and is described next.
3.1 Local Kemenization
- The authors i n troduce the notion of a locally Kemeny optimal aggregation, a relaxation of Kemeny optimality, that ensures satisfaction of the extended Condorcet principle and yet remains computationally tractable.
- The authors have discussed the value of the extended Condorcet criterion in increasing resistance to search engine spam and in ensuring that elements in the top partitions remain highly ranked.
- By applying their \local Kemenization" procedure (described below), one can obtain a ranking that is maximally consistent with the Borda ordering but in which the Condorcet winners are at the top of the list.the authors.
- Intuitively, this approach also preserves the strengths of the initial aggregation .
- Where the authors also show that the local Kemenization of an aggregation is unique.
4.2 Footrule and scaled footrule
- Since the footrule optimal aggregation is a good approximation of Kemeny optimal aggregation, it merits investigation.
- Now, the authors obtain an algorithm for footrule optimal aggregation via the following proposition: Proposition 4.
- It can be shown that a permutation minimizing the total footrule distance to the i's is given by a minimum cost perfect matching in the bipartite graph.
- As before, the authors can solve the minimum cost maximum matching problem on this bipartite graph to obtain the footrule aggregation algorithm for partial lists.
- The authors called this method the scaled footrule aggregation (SFO).
5.1 Meta-search
- Several meta-search engines exist (e.g., metacrawler 3]) and many W eb users build their own meta-search engines.
- Given the di erent c r a wling strategies, indexing policies, and ranking functions employed by di erent search engines, meta-search engines are useful in many situations.
- The actual success of a meta-search engine directly depends on the aggregation technique underlying it.
- Given a query, obtain the top (say) 100 results from many s e a r c h engines, apply the rank aggregation function with the universe being the union of pages returned by the search engines, and return the top (say) 100 results of the aggregation, also known as The idea is simple.
- The authors illustrate this scheme in Section 6.2.1 and examine the performance of their methods.
5.2 Aggregating ranking functions
- Given a collection of documents, the problem of indexing is: store the documents in such a manner that given a search term, those most relevant to the search term can be retrieved easily.
- Another ranking function might be the consequence of applying the vector-space model and an appropriate distance measure to the document collection.
- The authors techniques can be applied to obtain a good aggregation of these ranking functions.
- If the system is exible enough to let the user specify various preference criteria (travel dates/times, window/aisle seating, number of stops, frequent-ier preferences, refundable/non-refundable nature of ticket purchase, and of course, price), it can rank the available ight plans based on each of the criteria, and apply rank aggregation methods to give better quality results to the user.
- In fact, very often there is not even a clear order of importance among the criteria.
5.3 Spam reduction
- As the authors discussed earlier, the extended Condorcet principle is a reasonable cure for spam.
- This extra step is inexpensive in terms of computation cost, but has the bene t of reducing spam by ranking Condorcet losers below Condorcet winners.
5.4 Word association techniques
- Di erent search engines and portals have di erent semantics of handling a multi-word query.
- As discussed in Section 1.1, both these scenarios are inconvenient in many situations.
- The user lists a number of skills and a number of potential keywords in the job description, for example, "Silicon Valley C++ Java CORBA TCPIP algorithms start-up pre-IPO stock options".
- It is clear that the \AND" rule might produce no document, and the \OR" rule is equally disastrous.
- The authors query the search engine with these k sub-queries (using the AND semantics) and obtain k top d (say, d = 100) results for each o f the sub-queries.
5.5 Search engine comparison
- The authors methods also imply a natural way to compare the performance of various search engines.
- The main idea is that a search engine can be called good when it behaves like a least noisy expert for a query.
- This agrees with their earlier notion of what an expert is and how to deal with noisy experts.
- Thus, the procedure to rank the search engines themselves (with respect to a query) is as follows: obtain a rank aggregation of the results from various search engines and rank the search engines based on their (Kendall or footrule) distance to the aggregated ranking.
6.1 Infrastructure
- The rst experiment is to build a meta-search engine using di erent aggregation methods (Section 4) and compare their performances.
- The third experiment is to illustrate the technique of word association for multiword queries.
- While the authors provide numerical values for the rst experiment, they p r o vide actual examples for the second and third experiments.
- The authors distance measurements are with respect to union of the top 100 results from these search engines.
- The authors notion of two urls being identical is purely syntactic (up to some canonical form) the authors do not use the content of page to determine if two urls are identical.
6.2.1 Meta-search
- The fourth column in the table means that 27.231 pages (on average) were present in exactly three of the search engine results.
- The second column indicates that around 284 pages were present in only one search engine while the last column indicates that less than 2 pages were present in all the search engines.
- The performance is calculated in terms of the three distance measures described in Section 2.1.
6.2.2 Spam reduction
- The authors use the following queries: Feng Shui, organic vegetables, gardening.
- Notice that their de nition of spam does not mean evil!.
- On the other hand, the authors w ere interested in urls that spammed at least two search engines | given that the overlap among search engines was not very high, this proved to be a challenging task.
- Table 3 presents their examples: the entries are the rank within individual search engines' lists.
- Based on results from Section 6.2.1, the authors restrict their attention to SFO and MC4 with local Kemenization.
6.2.3 Word associations
- As noted earlier, Google uses AND semantics and hence for many i n teresting multi-word queries, the number or the quality o f the pages returned is not very high.
- The authors c hose every pair of terms in the multi-word query to construct several lists and the apply rank aggregation (SFO and MC4) t o these lists.
6.3 Discussion
- Of all the methods, MC4 outperforms all others.
- This is very interesting since Borda's method is the usual choice of aggregation, and perhaps the most natural.
- Recall that the footrule procedure for partial lists was only a heuristic modi cation of the footrule procedure for full lists.
- In general, local Kemenization seems to improve around 1{3% in terms of the distance measures.
- Examining the results in Section do not claim that their methods completely eliminate spam, their study shows that they reduce spam in general.
7. CONCLUSIONS AND FURTHER WORK
- The authors have developed the theoretical groundwork for describing and evaluating rank aggregation methods.
- The methods are also simple to implement, do not have a n y computational overhead, and out-perform popular classical like Borda's method.
- The authors h a ve established the value of the extended Condorcet criterion in the context of meta-search, and have described a simple process, local Kemenization, for ensuring satisfaction of this criterion.
- Further work involves trying to obtain a qualitative understanding of why the Markov c hain methods perform very well.
- Finally, this work originated in conversations with Helen Nissenbaum on bias in searching.
Did you find this useful? Give us your feedback
Citations
2,511 citations
1,813 citations
Cites background from "Rank aggregation methods for the We..."
...Intuitively, this merged top-N list should reflect the highest quality ranking possible, also known as the “rank aggregation problem” [6]....
[...]
1,765 citations
Cites background or methods from "Rank aggregation methods for the We..."
...For our experiments, we used 35 of the sample queries given in [9], which were in turn compiled from earlier papers....
[...]
...See [9] for a discussion of various distance measures for ranked lists in the context of Web search results....
[...]
1,548 citations
1,161 citations
Cites methods from "Rank aggregation methods for the We..."
...For our experiments, we used 35 of the sample queries given in [12], which were in turn compiled from earlier papers....
[...]
References
14,696 citations
"Rank aggregation methods for the We..." refers background or methods in this paper
...Page....
[...]
...Afewyearsago, querytermfrequencywasthesinglemainheuristicinrankingWebpages;sincetheinfuentialworkofKleinberg[16] andBrinandPage[7],linkanalysishascometobeidentifedasaverypowerfultechniqueinrankingWebpagesand otherhyperlinkeddocuments....
[...]
...Yet other ranking functions might be the ones implied by PageRank [7] and Clever [16, 8]....
[...]
...On the Web, such an approach has already proved tremendously successful [16, 8, 7]....
[...]
...the context of Web searching, the HITS algorithm of Kleinberg [16] and the PageRank algorithm of Brin and Page [7] are motivated by similar considerations....
[...]
13,327 citations
8,328 citations
3,571 citations
Additional excerpts
...(see [20])....
[...]
Related Papers (5)
Frequently Asked Questions (8)
Q2. What are the future works in "Rank aggregation methods for the web" ?
Further work involves trying to obtain a qualitative understanding of why the Markov chain methods perform very well. Also, it will be interesting to measure the e cacy of their methods on a document base with several competing ranking functions.
Q3. What is a strongly connected component of a graph?
A strongly connected component of a graph is a set of nodes such that for every pair of nodes in the component, there is a directed path from one to the other.
Q4. How can the authors make a ranking function satisfy the extended Condorcet principle?
Using the technique of local Kemenization, it is easy to take any rank aggregation method and tweak its output to make it satisfy the extended Condorcet principle.
Q5. How do you get the top results from the search engines?
The idea is simple: given a query, obtain the top (say) 100 results from many search engines, apply the rank aggregation function with the universe being the union of pages returned by the search engines, and return the top (say) 100 results of the aggregation.
Q6. What is the problem of constructing a good meta-search engine?
As the authors observed earlier, the problem of constructing a good meta-search engine is tantamount to obtaining a good rank aggregation function for partial and top d lists.
Q7. What is the motivation for studying rank aggregation in the context of the Web?
their rst motivation for studying rank aggregation in the context of the Web is to provide users a certain degree of robustness of search, in the face of various shortcomings and biases | malicious or otherwise | of individual search engines.
Q8. What other heuristics have been added to the list of Web applications?
Several other heuristics have been added, including anchor-text analysis [8], page structure (headers, etc.) analysis, the use of keyword listings and the url text itself, etc.