This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users The primary goal of this article is to catalog and compile material related to this topic into a single source This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR) Finally, this article concludes with a discussion of outstanding challenges and future research directions

https://www.nowpublishers.com/article/DownloadSummary/INR-012

Methods for Evaluating Interactive Information Retrieval Systems with Users

Data governance: A conceptual framework, structured review, and research agenda

One of the most challenging aspects of designing interactive information retrieval (IIR) experiments with users is the development of search tasks. We describe an evaluation of 20 search tasks that were designed for use in IIR experiments and developed using a cognitive complexity framework from educational theory. The search tasks represent five levels of cognitive complexity and four topical domains. The tasks were evaluated in the context of a laboratory IIR experiment with 48 participants. Behavioral and self-report data were used to characterize and understand differences among tasks. Results showed more cognitively complex tasks required significantly more search activity from participants (e.g., more queries, clicks, and time to complete). However, participants did not evaluate more cognitively complex tasks as more difficult and were equally satisfied with their performances across tasks. Our work makes four contributions: (1) it adds to what is known about the relationship among task, search behaviors and user experience; (2) it presents a framework for task creation and evaluation; (3) it provides tasks and questionnaires that can be reused by others and (4) it raises questions about findings and assumptions of many recent studies that only use behavioral signals from search logs as evidence for task difficulty and searcher satisfaction, as many of our results directly contradict these findings.

https://www.ils.unc.edu/~jarguell/KellyICTIR2015.pdf

Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework

The goal of this paper is to provide guidance to researchers investigating exploratory search behaviors and exploratory search systems. It focuses on the design of search tasks assigned in such studies. Based on a review of past studies, a set of task characteristics associated with exploratory search tasks are identified: exploratory search tasks focus on learning and investigative search goals; they are general (rather than specific), open-ended, and often target multiple items/documents; they involve uncertainty and are motivated by ill-defined or ill-structured problems; they are dynamic and evolve over time; they are multi-faceted and may be procedurally complex; and they are often accompanied by other information or cognitive behaviors, such as sensemaking. Recommendations are provided for the design of search task descriptions that will elicit exploratory search behaviors.

/pdf/assigning-search-tasks-designed-to-elicit-exploratory-search-4lypo97mnk.pdf

Assigning search tasks designed to elicit exploratory search behaviors

Looking for Information -- A Survey of Research on Information Seeking, Needs, and Behavior by Donald Case

The purpose of this research is to examine how search differs according to selected task variables. Three types of task information goals and two types of task structures were explored. This mixed within- and between-subjects designed study had 96 participants complete three of 12 tasks in a laboratory setting using a specialized search system based on Lucene. Using a combination metrics (user perception collected by questionnaires, transaction log data, and characteristics of relevant documents), we assessed the effect of goals and structure on search as demonstrated through queries and their use in interactive searching.

Task Effects on Interactive Search: The Query Factor

As the public sector increasingly transforms itself and its processes through digital technologies, information—the prime resource—the governance of information, and the need for advanced information skills, have moved to centre stage. Information governance processes form a critical bridge between the legacy systems of the past and the agile and mobile platforms of the future. Describing the forces at play, which include new information formats, modes of transmission and uses but also new skills and governance requirements, this article proposes an information governance-centred research agenda. Considering information governance through a holistic lens is essential to ensure that the new possibilities available through technology can be leveraged to transform government, while government information itself is preserved as the essential tool for current and future government decision-making, as a critical service to the public, as public memory, and as the foundation for public sector accountability and transparency.

https://onlinelibrary.wiley.com/doi/pdf/10.1111/capa.12227

Information governance in digitized public administration

Pandemic Victory Gardens: Potential for local land use policies

The Internet is increasingly seen as integral to economic progress and prosperity. Yet how the Internet will be managed as it grows and diversifies remains a hotly contested topic, as the debate on net neutrality demonstrates. Whether the Internet is neutral or not has serious implications for Internet service providers (ISPs), businesses operating online, governments, and civil society. With these stakeholders and varying interests at play, the debate about net neutrality is often characterized in terms of polar positions, and the discussion has seemed intransigent and ongoing with an uncertain end point. To increase understanding about the debate, this paper combines a review of the literature on net neutrality with evidence from interviews with four individuals, each representing the viewpoint of a major stakeholder group in Canada. Analysis of the similarities and differences among key stakeholder positions shows that in fact the positions are more complex and considerably more nuanced than typically depicted. By focussing on components of the issues, and staying away from the politics of contesting net neutrality, progress in the debate can be made. While this paper gives attention to the Canadian context in particular, the findings echo those of international organizations, and adds to the global conversation on the future of the Internet.

Sandra Toze

Papers

Task Effects on Interactive Search: The Query Factor

Information governance in digitized public administration

Pandemic Victory Gardens: Potential for local land use policies

Understanding the net neutrality debate: Listening to stakeholders

Examining Group Process through an Information Behaviour Lens: How Student Groups Work with Information to Accomplish Tasks