User engagement refers to the quality of the user experience that emphasizes the positive aspects of interacting with an online application and, in particular, the desire to use that application longer and repeatedly. User engagement is a key concept in the design of online applications (whether for desktop, tablet or mobile), motivated by the observation that successful applications are not just used, but are engaged with. Users invest time, attention, and emotion in their use of technology, and seek to satisfy pragmatic and hedonic needs. Measurement is critical for evaluating whether online applications are able to successfully engage users, and may inform the design of and use of applications. User engagement is a multifaceted, complex phenomenon; this gives rise to a number of potential measurement approaches. Common ways to evaluate user engagement include using self-report measures, e.g., questionnaires; observational methods, e.g. facial expression analysis, speech analysis; neuro-physiological signal processing methods, e.g., respiratory and cardiovascular accelerations and decelerations, muscle spasms; and web analytics, e.g., number of site visits, click depth. These methods represent various trade-offs in terms of the setting (laboratory versus ``in the wild''), object of measurement (user behaviour, affect or cognition) and scale of data collected. For instance, small-scale user studies are deep and rich, but limited in terms of generalizability, whereas large-scale web analytic studies are powerful but negate users' motivation and context. The focus of this book is how user engagement is currently being measured and various considerations for its measurement. Our goal is to leave readers with an appreciation of the various ways in which to measure user engagement, and their associated strengths and weaknesses. We emphasize the multifaceted nature of user engagement and the unique contextual constraints that come to bear upon attempts to measure engagement in different settings, and across different user groups and web domains. At the same time, this book advocates for the development of ``good'' measures and good measurement practices that will advance the study of user engagement and improve our understanding of this construct, which has become so vital in our wired world. Table of Contents: Preface / Acknowledgments / Introduction and Scope / Approaches Based on Self-Report Methods / Approaches Based on Physiological Measurements / Approaches Based on Web Analytics / Beyond Desktop, Single Site, and Single Task / Enhancing the Rigor of User Engagement Methods and Measures / Conclusions and Future Research Directions / Bibliography / Authors' Biographies / Index

Measuring User Engagement

This paper discusses the use of web metrics tools at four digital libraries, the Instructional Architect, the Library of Congress, the National Science Digital Library, and WGBH Teachers' Domain. We describe some of the issues involved in using web metrics to report on ongoing web site performance. We also describe how web metrics can be used for focused data mining, using session length metrics as our example; and here, we recommend that session length metrics, which were developed to track e-commerce, need to be carefully considered when they are applied in non-e-commerce settings, such as digital libraries. We conclude by discussing some of the current limitations and possibilities of using web metrics to analyze and evaluate digital library use and impact.

Using web metrics to analyze digital libraries

This article presents a quasi-experimental study comparing the impact of two technology-related teacher professional development (TTPD) designs, aimed at helping junior high school science and mathematics teachers design online activities using the rapidly growing set of online learning resources available on the Internet. The first TTPD design (tech-only) focused exclusively on enhancing technology knowledge and skills for finding, selecting, and designing classroom activities with online resources, while the second (tech + pbl) coupled technology knowledge with learning to design problem-based learning (PBL) activities for students. Both designs showed large pre-post gains for teacher participants (N = 36) in terms of self-reported knowledge, skills, and technology integration. Significant interaction effects show that teachers in the tech + pbl group had larger gains for self-reported knowledge and externally rated use of PBL. Three generalized estimating equation (GEE) models were fit to study the impact on students’ (N = 1,247) self reported gains in behavior, knowledge, and attitudes. In the resulting models, students of tech + pbl teachers showed significant increases in gain scores for all three outcomes. By contrast, students of tech-only teachers showed improved gains only in attitudes.

/pdf/comparing-technology-related-teacher-professional-5e5jdqv1if.pdf

Comparing technology-related teacher professional development designs: a multilevel study of teacher and student impacts

This article describes two consecutive enactments of technology-oriented teacher professional development designs, aimed at helping teachers find high-quality online learning resources and use them in designing effective problem-based learning (PBL) activities for their students. To align with current professional development prescriptions, in the first enactment, teachers learned PBL design skills concurrently with technology skills. Following aspects of design-based research, the professional development theory, participant feedback, and results from the first enactment informed the design of the second. In this second enactment, technology skills were separated and presented prior to learning about PBL. Results from a mixed-methods study of impact indicated that both professional development enactments were associated with large increases in teacher knowledge, experience, and confidence with regards to technology use and integration. Variations in the level of PBL usage by teachers in their activities, and the degree to which they discuss PBL and technology integration are presented alongside limitations, practical significance, scholarly significance, and planned future work.

/pdf/integrating-technology-and-problem-based-learning-a-mixed-3l64q4ivma.pdf

Integrating Technology and Problem-based Learning: A Mixed Methods Study of Two Teacher Professional Development Designs

This paper summarizes results from a national survey of 4,678 respondents, representing 119 institutions of higher education in the United States regarding their use of digital resources for scholarly purposes. This paper presents the following results: (1) demographics commonly used in higher education to categorize populations such as institution type or level of teaching experience could not reliably predict use of online digital resources, (2) valuing online digital resources corresponds with only higher levels of use for certain types of digital resources, (3) lack of time was a significant barrier to use of materials while, paradoxically, respondents indicated that they used them because they save time, (4) respondents did not tend to intentionally look to the Internet as a trusted resource for learning about teaching.

The use of online digital resources and educational digital libraries in higher education

We report preliminary lessons from a year of webmetrics research with two digital libraries Despite the apparent 'plug-and-play-and-report' nature of webmetrics tools, much work was required to extract useful data from the tools used

Robert A. Donahue

Papers

Using web metrics to analyze digital libraries

Evaluating digital libraries with webmetrics