Rigorous evidence identification is essential for systematic reviews and meta-analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query-based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.

https://onlinelibrary.wiley.com/doi/pdf/10.1002/jrsm.1378

Which Academic Search Systems are Suitable for Systematic Reviews or Meta‐Analyses? Evaluating Retrieval Qualities of Google Scholar, PubMed and 26 other Resources

The majority of academic papers are scarcely cited while a few others are highly cited. A large number of studies indicate that there are many factors influencing the number of citations. An actual review is missing that provides a comprehensive review of the factors predicting the frequency of citations. In this review, we performed a search in WoS, Scopus, PubMed and Medline to retrieve relevant papers. In overall, 2087 papers were retrieved among which 198 relevant papers were included in the study. Three general categories with twenty eight factors were identified to be related to the number of citations: Category one: "paper related factors": quality of paper; novelty and interest of subject; characteristics of fields and study topics; methodology; document type; study design; characteristics of results and discussion; use of figures and appendix in papers; characteristics of the titles and abstracts; characteristics of references; length of paper; age of paper; early citation and speed of citation; accessibility and visibility of papers. Category two: "journal related factors": journal impact factor; language of journal; scope of journal; form of publication. Category three: "author(s) related factors": number of authors; author's reputation; author's academic rank; self-citations; international and national collaboration of authors; authors' country; gender, age and race of authors; author's productivity; organizational features; and funding. Probably some factors such as the quality of the paper, journal impact factor, number of authors, visibility and international cooperation are stronger predictors for citations, than authors' gender, age and race; characteristics of results and discussion and so on.

Factors affecting number of citations: a comprehensive review of the literature

Information on the size of academic search engines and bibliographic databases (ASEBDs) is often outdated or entirely unavailable. Hence, it is difficult to assess the scope of specific databases, such as Google Scholar. While scientometric studies have estimated ASEBD sizes before, the methods employed were able to compare only a few databases. Consequently, there is no up-to-date comparative information on the sizes of popular ASEBDs. This study aims to fill this blind spot by providing a comparative picture of 12 of the most commonly used ASEBDs. In doing so, we build on and refine previous scientometric research by counting query hit data as an indicator of the number of accessible records. Iterative query optimization makes it possible to identify a maximum number of hits for most ASEBDs. The results were validated in terms of their capacity to assess database size by comparing them with official information on database sizes or previous scientometric studies. The queries used here are replicable, so size information can be updated quickly. The findings provide first-time size estimates of ProQuest and EbscoHost and indicate that Google Scholar’s size might have been underestimated so far by more than 50%. By our estimation Google Scholar, with 389 million records, is currently the most comprehensive academic search engine.

/pdf/google-scholar-to-overshadow-them-all-comparing-the-sizes-of-2w8d1cye78.pdf

Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases

To succeed in today's interconnected and complex world, workers need to be able to think systemically, creatively, and critically. Equipping K-16 students with these twenty-first-century competencies requires new thinking not only about what should be taught in school but also about how to develop valid assessments to measure and support these competencies. In Stealth Assessment, Valerie Shute and Matthew Ventura investigate an approach that embeds performance-based assessments in digital games. They argue that using well-designed games as vehicles to assess and support learning will help combat students' growing disengagement from school, provide dynamic and ongoing measures of learning processes and outcomes, and offer students opportunities to apply such complex competencies as creativity, problem solving, persistence, and collaboration. Embedding assessments within games provides a way to monitor players' progress toward targeted competencies and to use that information to support learning.Shute and Ventura discuss problems with such traditional assessment methods as multiple-choice questions, review evidence relating to digital games and learning, and illustrate the stealth-assessment approach with a set of assessments they are developing and embedding in the digital game Newton's Playground. These stealth assessments are intended to measure levels of creativity, persistence, and conceptual understanding of Newtonian physics during game play. Finally, they consider future research directions related to stealth assessment in education.

Stealth Assessment: Measuring and Supporting Learning in Video Games

Purpose – This paper aims to present guidelines for using streaming multimedia tutorials in academic library instruction.Design/methodology/approach – Research in cognitive psychology, education and librarianship is synthesized and its application to the design of library screencast or multimedia tutorials is discussed.Findings – Guidelines are presented on determining when multimedia is needed, how to make sure multimedia tutorials are clear and understandable, how to include interactivity effectively, how to promote critical thinking, and how to design interfaces and tutorial structure based on levels of student knowledge.Originality/value – Screencasts are a relatively new form of library instruction. Guidelines in this paper are summarized in a checklist that can be used when designing screencasts or multimedia tutorials.

/pdf/guidelines-for-effective-online-instruction-using-multimedia-4qleghcbue.pdf

Guidelines for effective online instruction using multimedia screencasts

Federated searching was once touted as the library world's answer to Google, but ten years since federated searching technology's inception, how does it actually compare? This study focuses on undergraduate student preferences and perceptions when doing research using both Google and a federated search tool. Students were asked about their preferences using each search tool and the perceived relevance of the sources they found using each search tool. Students were also asked to self-assess their online searching skills. The findings show that students believe they possess strong searching skills, are able to find relevant sources using both search tools, but actually prefer the federated search tool to Google for doing research. Thus, despite federated searching's limitations, students see the need for it, libraries should continue to offer federated search (especially if a discovery search tool is not available), and librarians should focus on teaching students how to use federated search and Google more effectively.

/pdf/google-vs-the-library-student-preferences-and-perceptions-1gadw3942h.pdf

Google vs. the Library: Student Preferences and Perceptions When Doing Research Using Google and a Federated Search Tool

This study examines the information-seeking behavior of undergraduate students within a research context. Student searches were recorded while the participants used Google and a library (federated) search tool to find sources (one book, two articles, and one other source of their choosing) for a selected topic. The undergraduates in this study believed themselves to be skilled researchers, but their search queries and behaviors did not support this belief. Students did not examine their topics to identify keywords and related terms. They relied heavily on the language presented to them via the list of research topics and performed natural language or simple keyword or phrase queries. They did not reformulate or refine their research questions or search queries, did not move beyond the first page of results, and did not examine metadata to improve searches. When using Google, students frequently visited commercial sites such as Amazon; content farms such as About.com; and subscription databases such as JSTOR. This study concludes by offering suggestions for search interface improvement and pedagogical opportunities on which librarians may wish to focus or refocus. This article is the second in a series that examines student use of Google and a library (federated) search tool.

https://academicworks.cuny.edu/cgi/viewcontent.cgi?article=1081&context=bc_pubs

Google vs. the Library (Part II): Student Search Patterns and Behaviors When Using Google and a Federated Search Tool.

Purpose – The purpose of the paper is to measure the effectiveness of a highly interactive animated tutorial that aims to teach basic information literacy concepts to undergraduates and to determine which factors enhance online learning.Design/methodology/approach – The tutorial was evaluated using 30 undergraduate students. Each test consisted of a pre‐tutorial questionnaire and a post‐tutorial questionnaire.Findings – Data from the evaluative testing of the tutorial shows that the tutorial is effective in imparting information literacy concepts to undergraduate students. Students responded positively to the interactivity and game‐like nature of the tutorial.Originality/value – The high degree of interactivity and the game‐like quality of the tutorial are key factors in its success. Thus, developers should aim to incorporate these elements into future online tutorials for undergraduate students.

Using interactive technology to teach information literacy concepts to undergraduate students

By analyzing the citation patterns of the linguistics literature, the authors provide a bibliometric description of the discipline that will help librarians who have reference, instruction, or collection development responsibilities in this area understand it better. One important aspect of such an understanding is determining where linguistics classifies within the humanities, the social sciences, and the sciences. Based on several of the citation patterns discovered, namely the importance of recent publications to the field, and the prominence of journals as a primary vehicle of scholarly communication, this analysis concludes that linguistics more closely resembles the disciplines of the social sciences.

A Citation Study of the Characteristics of the Linguistics Literature

This study assesses and compares the quality of sources found by undergraduate students when doing research using both Google and a library (federated) search tool. Thirty undergraduates were asked to find four sources (one book, two articles, and one additional source of their choosing) related to a selected research topic. Students used both Google and a federated search (resource discovery) tool to find material they believed to be relevant. Each source was evaluated for topic relevance, authority, appropriateness, and date, and assigned a total quality score. Results showed that the books found via Google were slightly higher quality than those uncovered via the federated search tool. The articles and additional sources students found via the federated search tool were slightly to moderately higher quality, respectively, than those discovered via Google.

/pdf/google-vs-the-library-part-iii-assessing-the-quality-of-3xv2dviy9f.pdf

Helen Georgas

Papers

Google vs. the Library: Student Preferences and Perceptions When Doing Research Using Google and a Federated Search Tool

Google vs. the Library (Part II): Student Search Patterns and Behaviors When Using Google and a Federated Search Tool.

Using interactive technology to teach information literacy concepts to undergraduate students

A Citation Study of the Characteristics of the Linguistics Literature

Google vs. the Library (Part III): Assessing the Quality of Sources Found by Undergraduates.