The Recommender Systems community is paying increasing attention to novelty and diversity as key qualities beyond accuracy in real recommendation scenarios. Despite the raise of interest and work on the topic in recent years, we find that a clear common methodological and conceptual ground for the evaluation of these dimensions is still to be consolidated. Different evaluation metrics have been reported in the literature but the precise relation, distinction or equivalence between them has not been explicitly studied. Furthermore, the metrics reported so far miss important properties such as taking into consideration the ranking of recommended items, or whether items are relevant or not, when assessing the novelty and diversity of recommendations.We present a formal framework for the definition of novelty and diversity metrics that unifies and generalizes several state of the art metrics. We identify three essential ground concepts at the roots of novelty and diversity: choice, discovery and relevance, upon which the framework is built. Item rank and relevance are introduced through a probabilistic recommendation browsing model, building upon the same three basic concepts. Based on the combination of ground elements, and the assumptions of the browsing model, different metrics and variants unfold. We report experimental observations which validate and illustrate the properties of the proposed metrics.

/pdf/rank-and-relevance-in-novelty-and-diversity-metrics-for-3zil1r0hk0.pdf

Rank and relevance in novelty and diversity metrics for recommender systems

With the rapid growth of web search in recent years the problem of modeling its users has started to attract more and more attention of the information retrieval community. This has several motivations. By building a model of user behavior we are essentially developing a better understanding of a user, which ultimately helps us to deliver a better search experience. A model of user behavior can also be used as a predictive device for non-observed items such as document relevance, which makes it useful for improving search result ranking. Finally, in many situations experimenting with real users is just infeasible and hence user simulations based on accurate models play an essential role in understanding the implications of algorithmic changes to search engine results or presentation changes to the search engine result page. In this survey we summarize advances in modeling user click behavior on a web search engine result page. We present simple click models as well as more complex models aimed at capturing non-trivial user behavior patterns on modern search engine result pages. We discuss how these models compare to each other, what challenges they have, and what ways there are to address these challenges. We also study the problem of evaluating click models and discuss the main applications of click models.

Click Models for Web Search

Novelty and diversity have been identified, along with accuracy, as foremost properties of useful recommendations. Considerable progress has been made in the field in terms of the definition of methods to enhance such properties, as well as methodologies and metrics to assess how well such methods work. In this chapter we give an overview of the main contributions to this area in the field of recommender systems, and seek to relate them together in a unified view, analyzing the common elements underneath the different forms under which novelty and diversity have been addressed, and identifying connections to closely related work on diversity in other fields.

Novelty and Diversity in Recommender Systems

An "Interestingness Modeler" uses deep neural networks to learn deep semantic models (DSM) of "interestingness." The DSM, consisting of two branches of deep neural networks or their convolutional versions, identifies and predicts target documents that would interest users reading source documents. The learned model observes, identifies, and detects naturally occurring signals of interestingness in click transitions between source and target documents derived from web browser logs. Interestingness is modeled with deep neural networks that map source-target document pairs to feature vectors in a latent space, trained on document transitions in view of a "context" and optional "focus" of source and target documents. Network parameters are learned to minimize distances between source documents and their corresponding "interesting" targets in that space. The resulting interestingness model has applicable uses, including, but not limited to, contextual entity searches, automatic text highlighting, prefetching documents of likely interest, automated content recommendation, automated advertisement placement, etc.

/pdf/modeling-interestingness-with-deep-neural-networks-1327539pqd.pdf

Modeling Interestingness with Deep Neural Networks

Understanding user browsing behavior in web search is key to improving web search effectiveness. Many click models have been proposed to explain or predict user clicks on search engine results. They are based on the probabilistic graphical model (PGM) framework, in which user behavior is represented as a sequence of observable and hidden events. The PGM framework provides a mathematically solid way to reason about a set of events given some information about other events. But the structure of the dependencies between the events has to be set manually. Different click models use different hand-crafted sets of dependencies. We propose an alternative based on the idea of distributed representations: to represent the user's information need and the information available to the user with a vector state. The components of the vector state are learned to represent concepts that are useful for modeling user behavior. And user behavior is modeled as a sequence of vector states associated with a query session: the vector state is initialized with a query, and then iteratively updated based on information about interactions with the search engine results. This approach allows us to directly understand user browsing behavior from click-through data, i.e., without the need for a predefined set of rules as is customary for PGM-based click models. We illustrate our approach using a set of neural click models. Our experimental results show that the neural click model that uses the same training data as traditional PGM-based click models, has better performance on the click prediction task (i.e., predicting user click on search engine results) and the relevance prediction task (i.e., ranking documents by their relevance to a query). An analysis of the best performing neural click model shows that it learns similar concepts to those used in traditional click models, and that it also learns other concepts that cannot be designed manually.

https://staff.fnwi.uva.nl/m.derijke/wp-content/papercite-data/pdf/borisov-neural-2016.pdf

A Neural Click Model for Web Search

Modeling a user's click-through behavior in click logs is a challenging task due to the well-known position bias problem. Recent advances in click models have adopted the examination hypothesis which distinguishes document relevance from position bias. In this paper, we revisit the examination hypothesis and observe that user clicks cannot be completely explained by relevance and position bias. Specifically, users with different search intents may submit the same query to the search engine but expect different search results. Thus, there might be a bias between user search intent and the query formulated by the user, which can lead to the diversity in user clicks. This bias has not been considered in previous works such as UBM, DBN and CCM. In this paper, we propose a new intent hypothesis as a complement to the examination hypothesis. This hypothesis is used to characterize the bias between the user search intent and the query in each search session. This hypothesis is very general and can be applied to most of the existing click models to improve their capacities in learning unbiased relevance. Experimental results demonstrate that after adopting the intent hypothesis, click models can better interpret user clicks and achieve a significant NDCG improvement.

/pdf/characterizing-search-intent-diversity-into-click-models-3hryf2uffl.pdf

Characterizing search intent diversity into click models

Click modeling aims to interpret the users' search click data in order to predict their clicking behavior. Existing models can well characterize the position bias of documents and snippets in relation to users' mainstream click behavior. Yet, current advances depict users' search actions only in a general setting by implicitly assuming that all users act in the same way, regardless of the fact that anyone, motivated with some individual interest, is more likely to click on a link than others. It is in light of this that we put forward a novel personalized click model to describe the user-oriented click preferences, which applies and extends matrix / tensor factorization from the view of collaborative filtering to connect users, queries and documents together. Our model serves as a generalized personalization framework that can be incorporated to the previously proposed click models and, in many cases, to their future extensions. Despite the sparsity of search click data, our personalized model demonstrates its advantage over the best click models previously discussed in the Web-search literature, supported by our large-scale experiments on a real dataset. A delightful bonus is the model's ability to gain insights into queries and documents through latent feature vectors, and hence to handle rare and even new query-document pairs much better than previous click models.

Personalized click model through collaborative filtering

High‐performance, fully atomically dispersed single‐atom catalysts (SACs) are promising candidates for next‐generation industrial catalysts. However, it remains a great challenge to avoid the aggregation of isolated atoms into nanoparticles during the preparation and application of SACs. Here, the evolution of Pd species is investigated on different crystal facets of CeO2, and vastly different behaviors on the single‐atomic dispersion of surface Pd atoms are surprisingly discovered. In situ X‐ray photoelectron spectroscopy (XPS), in situ near‐ambient‐pressure‐XPS (NAP‐XPS), in situ diffuse reflectance infrared Fourier transform spectroscopy (DRIFTS), and X‐ray absorption spectroscopy (XAS) reveal that, in a reducing atmosphere, more oxygen vacancies are generated on the (100) facet of CeO2, and Pd atoms can be trapped and thus feature atomic dispersion; by contrast, on the CeO2 (111) facet, Pd atoms will readily aggregate into clusters (Pdn). Furthermore, Pd1/CeO2(100) gives a high selectivity of 90.3% for the catalytic N‐alkylation reaction, which is 2.8 times higher than that for Pdn/CeO2(111). This direct evidence demonstrates the crucial role of crystal‐facet effects in the preparation of metal‐atom‐on‐metal‐oxide SACs. This work thus opens an avenue for the rational design and targeted synthesis of ultrastable and sinter‐resistant SACs.

Distinct Crystal‐Facet‐Dependent Behaviors for Single‐Atom Palladium‐On‐Ceria Catalysts: Enhanced Stabilization and Catalytic Properties

Recent advances in click models have positioned them as an effective approach to the improvement of interpreting click data, and some typical works include UBM, DBN, CCM, etc. After formulating the knowledge of user search behavior into a set of model assumptions, each click model developed an inference method to estimate its parameters. The inference method plays a critical role in terms of accuracy in interpreting clicks, and we observe that different inference methods for a click model can lead to significant accuracy differences. In this paper, we propose a novel Bayesian inference approach for click models. This approach regards click model under a unified framework, which has the following characteristics and advantages: 1. This approach can be widely applied to existing click models, and we demonstrate how to infer DBN, CCM and UBM through it. This novel inference method is based on the Bayesian framework which is more flexible in characterizing the uncertainty in clicks and brings higher generalization abilities. As a result, it not only excels in the inference methods originally developed in click models, but also provides a valid comparison among different models; 2. In contrast to the previous click models, which are exclusively designed for the position-bias, this approach is capable of capturing more sophisticated information such as BM25 and PageRank score into click models. This makes these models interpret click-through data more accurately. Experimental results illustrate that the click models integrated with more information can achieve significantly better performance on click perplexity and search ranking; 3. Because of the incremental nature of the Bayesian learning, this approach is scalable to process large scale and constantly growing log data.

Learning click models via probit bayesian inference

Atomically-precise preparation and atomic-level understanding on the single-atomic active sites with unique coordination structures in electrocatalysts for CO2 reduction reaction (CO2RR) remain a challenge. Here, we report a strategy of...

Botao Hu

Papers

Characterizing search intent diversity into click models

Personalized click model through collaborative filtering

Distinct Crystal‐Facet‐Dependent Behaviors for Single‐Atom Palladium‐On‐Ceria Catalysts: Enhanced Stabilization and Catalytic Properties

Learning click models via probit bayesian inference

Atomic-Level Engineering Fe1N2O2 Interfacial Structure Derived from Oxygen-Abundant Metal–Organic Frameworks to Promote Electrochemical CO2 Reduction