Objectives: To describe techniques for retrieval and appraisal used by consumers when they search for health information on the internet. Design: Qualitative study using focus groups, naturalistic observation of consumers searching the world wide web in a usability laboratory, and in-depth interviews. Participants: A total of 21 users of the internet participated in three focus group sessions. 17 participants were given a series of health questions and observed in a usability laboratory setting while retrieving health information from the web; this was followed by in-depth interviews. Setting: Heidelberg, Germany. Results: Although their search technique was often suboptimal, internet users successfully found health information to answer questions in an average of 5 minutes 42 seconds (median 4 minutes 18 seconds) per question. Participants in focus groups said that when assessing the credibility of a website they primarily looked for the source, a professional design, a scientific or official touch, language, and ease of use. However, in the observational study, no participants checked any “about us” sections of websites, disclaimers, or disclosure statements. In the post-search interviews, it emerged that very few participants had noticed and remembered which websites they had retrieved information from. Conclusions: Further observational studies are needed to design and evaluate educational and technological innovations for guiding consumers to high quality health information on the web. What is already known on this topic Little is known about how consumers retrieve and assess the quality of health information on the internet Qualitative data are needed to design educational and technological innovations to guide consumers to high quality health information What this study adds Users of the internet explore only the first few links on general search engines when seeking health information Consumers say that when assessing the credibility of a site they primarily look for the source, a professional design, and a variety of other criteria In practice, internet users do not check the “about us” sections of websites, try to find out who authors or owners of the site are, or read disclaimers or disclosure statements Very few internet users later remember from which websites they retrieved information or who stood behind the sites

https://www.bmj.com/content/bmj/324/7337/573.full.pdf

How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews

Word-of-mouth (WOM) study is extended to the on-line context (eWOM) by examining the informational and normative determinants of the perceived credibility of on-line consumer recommendations. A survey of users of an on-line consumer discussion forum in China substantiated the effects of the determinants, although post-hoc analyses revealed that prior knowledge and involvement level moderate some of them. Implications for research and practice are discussed.

Credibility of Electronic Word-of-Mouth: Informational and Normative Determinants of On-line Consumer Recommendations

As information becomes more ubiquitous and the demands that searchers have on search systems grow, there is a need to support search behaviors beyond simple lookup. Information seeking is the process or activity of attempting to obtain information in both human and technological contexts. Exploratory search describes an information-seeking problem context that is open-ended, persistent, and multifaceted, and information-seeking processes that are opportunistic, iterative, and multitactical. Exploratory searchers aim to solve complex problems and develop enhanced mental capacities. Exploratory search systems support this through symbiotic human-machine relationships that provide guidance in exploring unfamiliar information landscapes. Exploratory search has gained prominence in recent years. There is an increased interest from the information retrieval, information science, and human-computer interaction communities in moving beyond the traditional turn-taking interaction model support d by major Web search engines, and toward support for human intelligence amplification and information use. In this lecture, we introduce exploratory search, relate it to relevant extant research, outline the features of exploratory search systems, discuss the evaluation of these systems, and suggest some future directions for supporting exploratory search. Exploratory search is a new frontier in the search domain and is becoming increasingly important in shaping our future world. Table of Contents: Introduction / Defining Exploratory Search / Related Work / Features of Exploratory Search Systems / Evaluation of Exploratory Search Systems / Future Directions and concluding Remarks

Exploratory Search:Beyond the Query-Response Paradigm

In the Web, making judgments of information quality and authority is a difficult task for most users because overall, there is no quality control mechanism. This study examines the problem of the judgment of information quality and cognitive authority by observing people's searching behavior in the Web. Its purpose is to understand the various factors that influence people's judgment of quality and authority in the Web, and the effects of those judgments on selection behaviors. Fifteen scholars from diverse disciplines participated, and data were collected combining verbal protocols during the searches, search logs, and postsearch interviews. It was found that the subjects made two distinct kinds of judgment: predictive judgment, and evaluative judgment. The factors influencing each judgment of quality and authority were identified in terms of characteristics of information objects, characteristics of sources, knowledge, situation, ranking in search output, and general assumption. Implications for Web design that will effectively support people's judgments of quality and authority are also discussed.

/pdf/judgement-of-information-quality-and-cognitive-authority-in-17ekf2rxg2.pdf

Judgement of information quality and cognitive authority in the Web

In the last 16 years, more than 200 research articles were published about research-paper recommender systems. We reviewed these articles and present some descriptive statistics in this paper, as well as a discussion about the major advancements and shortcomings and an overview of the most common recommendation concepts and approaches. We found that more than half of the recommendation approaches applied content-based filtering (55 %). Collaborative filtering was applied by only 18 % of the reviewed approaches, and graph-based recommendations by 16 %. Other recommendation concepts included stereotyping, item-centric recommendations, and hybrid recommendations. The content-based filtering approaches mainly utilized papers that the users had authored, tagged, browsed, or downloaded. TF-IDF was the most frequently applied weighting scheme. In addition to simple terms, n-grams, topics, and citations were utilized to model users' information needs. Our review revealed some shortcomings of the current research. First, it remains unclear which recommendation concepts and approaches are the most promising. For instance, researchers reported different results on the performance of content-based and collaborative filtering. Sometimes content-based filtering performed better than collaborative filtering and sometimes it performed worse. We identified three potential reasons for the ambiguity of the results. (A) Several evaluations had limitations. They were based on strongly pruned datasets, few participants in user studies, or did not use appropriate baselines. (B) Some authors provided little information about their algorithms, which makes it difficult to re-implement the approaches. Consequently, researchers use different implementations of the same recommendations approaches, which might lead to variations in the results. (C) We speculated that minor variations in datasets, algorithms, or user populations inevitably lead to strong variations in the performance of the approaches. Hence, finding the most promising approaches is a challenge. As a second limitation, we noted that many authors neglected to take into account factors other than accuracy, for example overall user satisfaction. In addition, most approaches (81 %) neglected the user-modeling process and did not infer information automatically but let users provide keywords, text snippets, or a single paper as input. Information on runtime was provided for 10 % of the approaches. Finally, few research papers had an impact on research-paper recommender systems in practice. We also identified a lack of authority and long-term research interest in the field: 73 % of the authors published no more than one paper on research-paper recommender systems, and there was little cooperation among different co-author groups. We concluded that several actions could improve the research landscape: developing a common evaluation framework, agreement on the information to include in research papers, a stronger focus on non-accuracy aspects and user modeling, a platform for researchers to exchange information, and an open-source framework that bundles the available recommendation approaches.

/pdf/research-paper-recommender-systems-a-literature-survey-3o8izjcp9m.pdf

Research-paper recommender systems: a literature survey

IR research has a strong tradition of laboratory evaluation of systems. Such research is based on test collections, pre-defined test topics, and standard evaluation metrics. While recent research has emphasized the user viewpoint by proposing user-based metrics and non-binary relevance assessments, the methods are insufficient for truly user-based evaluation. The common assumption of a single query per topic and session poorly represents real life. On the other hand, one well-known metric for multiple queries per session, instance recall, does not capture early (within session) retrieval of (highly) relevant documents. We propose an extension to the Discounted Cumulated Gain (DCG) metric, the Session-based DCG (sDCG) metric for evaluation scenarios involving multiple query sessions, graded relevance assessments, and open-ended user effort including decisions to stop searching. The sDCG metric discounts relevant results from later queries within a session. We exemplify the sDCG metric with data from an interactive experiment, we discuss how the metric might be applied, and we present research questions for which the metric is helpful.

Discounted cumulated gain based evaluation of multiple-query IR sessions

OBJECTIVES: Assess query expansion using thesaurus relationships and definitions in the UMLS Metathesaurus for improving searching performance. METHODS: The queries from a MEDLINE test collection (OHSUMED) were expanded using synonym, hierarchical, and related term information as well as term definitions from the UMLS Metathesaurus. Documents were retrieved from a word-statistical retrieval system and assessed for recall and precision based on relevance judgments from the test collection. RESULTS: All types of query expansion degraded aggregate retrieval performance as measured by recall and precision, although 38.6% of the queries with synonym expansion and up to 29.7% of the queries with hierarchical expansion showed improvement. CONCLUSIONS: Thesaurus-based query expansion causes a decline in retrieval performance generally but improves it in specific instances. Further research must focus on identifying instances where performance improves and how it can be exploited by real users.

Assessing thesaurus-based query expansion using the UMLS Metathesaurus.

Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

Do batch and user evaluations give the same results

The World Wide Web is an increasingly popular source for consumer health information, but many authors have expressed concerns about the quality of health information present on the Internet. We have developed a prototype system that responds to a consumer health query by returning a list of Web pages that are ranked according to the likely quality of the page contents. A computer program identifies some of the criteria that have been suggested for assessing the quality of health information on the Internet. It also identifies characteristics that may serve as proxies for desirable (or undesirable) qualities that are difficult to assess directly using an algorithm. Intervening in the search process and automatically analyzing the contents of each page returned by a general search engine may facilitate the search for high quality consumer health information on the Web.

Filtering Web pages for quality indicators: an empirical approach to finding high quality consumer health information on the World Wide Web.

The SQL Server 11 release (code named "Denali") introduces a new data warehouse query acceleration feature based on a new index type called a column store index. The new index type combined with new query operators processing batches of rows greatly improves data warehouse query performance: in some cases by hundreds of times and routinely a tenfold speedup for a broad range of decision support queries. Column store indexes are fully integrated with the rest of the system, including query processing and optimization. This paper gives an overview of the design and implementation of column store indexes including enhancements to query processing and query optimization to take full advantage of the new indexes. The resulting performance improvements are illustrated by a number of example queries.

Susan Price

Papers

Discounted cumulated gain based evaluation of multiple-query IR sessions

Assessing thesaurus-based query expansion using the UMLS Metathesaurus.

Do batch and user evaluations give the same results

Filtering Web pages for quality indicators: an empirical approach to finding high quality consumer health information on the World Wide Web.

SQL server column store indexes