Creativity: Flow and the Psychology of Discovery and Invention

Cognitive engineering

Observing Interaction: An Introduction to Sequential Analysis, Roger Bakeman, John M. Gottman. Cambridge University Press, Cambridge (1986), ixv, +221. Price £25.00 hardback, £7.95 paperback

In this paper we report on an initial survey of user evaluation techniques used in Augmented Reality (AR) research. To identify all papers which include AR evaluations we reviewed research publications between the years 1993 and 2007 from online databases of selected scientific publishers. Starting with a total of 6071 publications we filtered the articles in several steps which resulted in 165 AR related publications with user evaluations. These publications were classified in two different ways: according to the evaluation type used following an earlier literature survey classification scheme; and according to the evaluation methods or approach used. We present the results of out literature survey, provide a comprehensive list of references of the selected publications, and discuss some possible research opportunities for future work. Keywords: Augmented Reality, user evaluation, evaluation methods. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for non-profit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of

A survey of evaluation techniques used in augmented reality studies

International Journal of Mobile Computing and Multimedia Communications: Guest editorial preface

This article describes an experiment comparing three Usability Evaluation Methods: User Testing (UT), Document-based Inspection (DI), and Expert Inspection (EI) for evaluating Virtual Environments (VEs). Twenty-nine individuals (10 end-users and 19 junior usability experts) participated during 1 hr each in the evaluation of two VEs (a training VE and a 3D map). Quantitative results of the comparison show that the effectiveness of UT and DI is significantly better than the effectiveness of EI. For each method, results show their problem coverage: DI- and UT-based diagnoses lead to more problem diversity than EI. The overlap of identified problems amounts to 22% between UT and DI, 20% between DI and EI, and 12% between EI and UT for both virtual environments. The identification impact of the whole set of usability problems is 60% for DI, 57% for UT, and only 36% for EI for both virtual environments. Also reliability of UT and DI is significantly better than reliability of EI. In addition, a qualitative anal...

Comparing Inspections and User Testing for the Evaluation of Virtual Environments

This paper introduces a new version of Ergonomic Criteria for HVEIs which have been tested for validity. An assignment experiment (Ergonomic Criteria X Usability Problems) has been conducted to ensure the appropriate understanding of the different criteria. The results of this study identify the criteria that are well understood and the ones that need further improvement.

Adaptation of Ergonomic Criteria to Human-Virtual Environments Interactions.

The goal of this paper is to survey the main issues with the ergonomic evaluation of MRS (Mixed Reality Systems) and to stimulate discussions for future research. A first point concerns definitions and specificities of MRS within the « reality / virtuality » continuum, and the incorporation of user issues. Another point concerns the combinatory character of the ergonomic knowledge to be applied to MRS entities (reality, « GUIs », « VR », and « MR » specific). A major issue concerns ergonomic evaluation methods, their current state, their advantages and drawbacks, particularly for user testing. Finally, the discussion points at various items which may be part of a future research agenda, such as the need for more usability data, for generic and well controlled experiments; for common testing platforms, for shared recommendations data bases, for design and assessment of inspection methods, for common task taxonomies and for common models of MR entities and situations, etc.; all of this possibly leading to increased knowledge based on shared benchmarks.

/pdf/obstacles-and-perspectives-for-evaluating-mixed-reality-4zrbb8nkgw.pdf

Obstacles and Perspectives for Evaluating Mixed Reality Usability.

Research problem: Despite the increasing interests raised by incident reporting systems, it is still unclear what dimensions of user experience (UX) and other contextual factors should be taken into account for the various stages of declaring an incident using mobile-phone applications. Research questions: How do citizens perceive and describe urban incidents? What UX dimensions are important for reporting an incident with a mobile-phone application? What other (contextual) factors are important from the users' point of view when declaring incidents? Which of the UX dimensions and contextual factors are important when in the various phases during an incident declaration? Literature review: Overall, there is a lack of empirical research in the domain of incident reporting. In general, the UX dimensions-visual and aesthetic experience, emotion, stimulation, identification, meaning and value, and social relatedness/coexperience-are important when designing interactive systems. It also shows that incidents are related to the citizen's perception of the environment. Methodology: A triangulated method approach combining interviews, a survey of existing systems, and a model-based task analysis were applied. This allows us to present a generic task model for incident reporting with a detailed description of UX dimensions affected in the various subtasks. Results and conclusions: Our findings point out the effect of UX dimensions in the task engaged by users when reporting urban citizens. The overall UX is directly influenced by the perceived level of severity, inconvenience and involvement, the personal context, and the technological mobile context. We have found that while several UX dimensions are highly relevant, they are not equally distributed along the several subtasks that citizens engage when reporting incidents. This study shows that semistructured requirement interviews can provide information about UX dimensions and it highlights the importance of the identification of UX dimensions in early phases of the development process.

/pdf/identifying-user-experience-dimensions-for-mobile-incident-5gkl8s6k0a.pdf

Identifying User Experience Dimensions for Mobile Incident Reporting in Urban Contexts

This paper introduces a version of Ergonomic Criteria adapted to HVEIs (Human Virtual Environment Interactions) which have been tested for validity. An experiment based on an assignment task (Ergonomic Criteria to usability problems) has been conducted to ensure appropriate understanding of the different criteria. The results of this study identify criteria that are clearly understood and criteria that need further improvement.

Cédric Bach

Papers

Comparing Inspections and User Testing for the Evaluation of Virtual Environments

Adaptation of Ergonomic Criteria to Human-Virtual Environments Interactions.

Obstacles and Perspectives for Evaluating Mixed Reality Usability.

Identifying User Experience Dimensions for Mobile Incident Reporting in Urban Contexts

Ergonomic criteria adapted to human virtual environment interaction