Supporting novice usability practitioners with usability engineering tools

TLDR: This work introduces a tool feature, usability problem instance records, to better support novice usability practitioners and describes the results of a study of this feature, which suggest that the feature helps to improve two aspects of the effectiveness of novice usability practitioner: reliability and quality.

Abstract: Although usability practitioners have successfully applied usability engineering processes to increase the usability of interaction designs, the literature suggests that usability practitioners experience a number of difficulties that negatively impact the effectiveness of their work, which in turn affects the effectiveness of the usability engineering processes within which they work. These difficulties include identifying and recording critical usability data and understanding and establishing relationships among usability data. These difficulties are particularly pronounced for novice usability practitioners. One approach to addressing these difficulties is through appropriate usability engineering tool support. We argue that existing usability engineering tools offer excellent support for helping experienced usability practitioners more efficiently do their jobs, but they provide little support for helping novice usability practitioners more effectively do their jobs. We introduce a tool feature, usability problem instance records, to better support novice usability practitioners. We describe the results of a study of this feature, which suggest that the feature helps to improve two aspects of the effectiveness of novice usability practitioners: reliability and quality.

Figures

Table 7: Results of the second Wizard study

Table 38: Study 3 - Time value by treatment, descriptive statistics

Table 6: Second version of the Wizard

Table 46: Study 3 - Quality as rated by developers, simple effects due to question for each treatment explored using slices

Figure 27: An expanding list that contains a group record

Figure 20: Node detail view

Full text

Supporting Novice Usability Practitioners with
Usability Engineering Tools
Jonathan Randall Howarth
Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State
University in partial fulfillment of the requirements for the degree of
Doctor of Philosophy
in
Computer Science and Applications
Dr. Rex Hartson, Virginia Tech, Chair
Dr. Tonya Smith-Jackson, Virginia Tech
Dr. Manuel Pérez-Quiñones, Virginia Tech
Dr. Terence Andre, United States Air Force Academy
Dr. Andrea Kavanaugh, Virginia Tech
April 13, 2007
Blacksburg, Virginia
Keywords: Usability Engineering, Usability Engineering Tool Support, Usability
Evaluation, Usability Problem Instances
Copyright 2007, Jonathan Randall Howarth

Supporting Novice Usability Practitioners with
Usability Engineering Tools
Jonathan Randall Howarth
Abstract
The usability of an application often plays an important role in determining its
success. Accordingly, organizations that develop software have realized the
need to integrate usability engineering into their development lifecycles. Although
usability practitioners have successfully applied usability engineering processes
to increase the usability of user-interaction designs, the literature suggests that
usability practitioners experience a number of difficulties that negatively impact
their effectiveness. These difficulties include identifying and recording critical
usability data, understanding and relating usability data, and communicating
usability information. These difficulties are particularly pronounced for novice
usability practitioners.
With this dissertation, I explored approaches to address these difficulties through
tool support for novice usability practitioners. Through an analysis of features
provided by existing tools with respect to documented difficulties, I determined a
set of desirable tool features including usability problem instance records,
usability problem diagnosis, and a structured process for combining and
associating usability problem data. I developed a usability engineering tool, the
Data Collection, Analysis, and Reporting Tool (DCART), which contains these
desirable tool features, and used it as a platform for studies of how these
desirable features address the documented difficulties.
The results of the studies suggest that appropriate tool support can improve the
effectiveness with which novice usability practitioners perform usability
evaluations. More specifically, tool support for usability problem instance records
helped novice usability practitioners more reliably identify and better describe
instances of usability problems experienced by participants. Additionally, tool
support for a structured process for combining and associating usability data
helped novice usability practitioners create usability evaluation reports that were
of higher quality as rated by usability practitioners and developers.
The results highlight key contributions of this dissertation, showing how tools can
support usability practitioners. They demonstrate the value of a structured
process for transforming raw usability data into usability information based on
usability problem instances. Additionally, they show that appropriate tool support
is a mechanism for further integrating usability engineering into the overall
software development lifecycle; tool support addresses the documented need for
more usability practitioners by helping novices perform more like experts.

iii
Dedication
I dedicate this dissertation to my family. I love you Mom, Dad, and Liz. Thanks for
all your support.
Additionally, I dedicate this dissertation to my friends who encouraged me and
occasionally made me tasty home-cooked meals to help keep me going during
late nights of work.

iv
Acknowledgements
I would to thank the members of my committee for their time and effort: Dr. Rex
Hartson, Dr. Tonya Smith-Jackson, Dr. Manuel Pérez-Quiñones, Dr. Terence
Andre, and Dr. Andrea Kavanaugh. In particular, I would to thank Dr. Rex
Hartson, my mentor and friend, for his input and guidance on all aspects of this
work and for the countless hours that he spent “word smithing” this dissertation.
I owe a special thanks to Dr. Miranda Capra for the many conversations that I
had with her concerning the design of my studies and the analysis and
interpretation of the results.
I also owe a special thanks to Pardha Pyla for many in-depth talks about usability
and the universe in general.
Additionally, I would like to thank Dr. Miranda Capra, Laurian Vega, Pardha Pyla,
and Dr. Beth Yost for the work that they did helping me collect data for my
studies. Each one contributed at least 20 hours of his or her time.
I would also like to thank the Air Force Office of Scientific Research for its
support with funding for a Small Business Technology Transfer Topic grant and
Dr. Randall Sparks of Knowledge Analysis Technologies with whom Dr. Rex
Hartson and I collaborated on the grant. The ideas for this dissertation developed
in part from work done on this grant.

v
Table of Contents
1 Introduction............................................................................................................................ 1
1.1 Motivation......................................................................................................................... 1
1.2 Problem............................................................................................................................ 1
1.3 Background......................................................................................................................1
1.3.1 Abstract Representation of Effectiveness ............................................................................... 1
1.3.2 Usability Engineering Process ................................................................................................ 2
1.3.3 Usability Evaluation Sub-Process ........................................................................................... 4
1.3.4 Formative Usability Evaluations.............................................................................................. 6
1.3.5 Usability Engineering Tool Support......................................................................................... 6
1.3.6 Usability Practitioner Skill........................................................................................................ 7
1.4 Scope ............................................................................................................................... 7
1.5 Research Goals, Research Questions, and Approach.................................................... 7
1.6 Contribution of Research ................................................................................................. 9
2 Related Work........................................................................................................................ 11
2.1 Difficulties Experienced by Usability Practitioners......................................................... 11
2.1.1 Evaluator Effect .................................................................................................................... 11
2.1.2 Content of Usability Problem Descriptions............................................................................ 11
2.1.3 Content of Usability Evaluation Reports................................................................................ 13
2.2 Existing Usability Engineering Tools.............................................................................. 14
2.2.1 Tools not Included in the Survey........................................................................................... 14
2.2.2 Categorization Scheme......................................................................................................... 15
2.2.3 Tools Included in the Survey................................................................................................. 16
2.3 User Action Framework ................................................................................................. 22
2.3.1 The Interaction Cycle and the User Action Framework......................................................... 22
2.3.2 Norman’s Model.................................................................................................................... 23
2.3.3 Evaluations of the UAF ......................................................................................................... 24
2.4 Determining the Need for Micro-Iteration....................................................................... 25
2.4.1 Exploratory Studies............................................................................................................... 26
2.4.2 Analogy to Medical Diagnosis............................................................................................... 27
2.5 Formative Studies of the Wizard.................................................................................... 29
3 Current State of Usability Engineering Tool Support ...................................................... 35
3.1 Categorization of Difficulties .......................................................................................... 35
3.1.1 Identifying and Recording Critical Usability Data .................................................................. 35
3.1.2 Understanding and Relating Usability Data .......................................................................... 36
3.1.3 Communicating Usability Information.................................................................................... 36
3.2 Existing Usability Engineering Tool Features ................................................................ 36
3.2.1 Low-Level Data Capture ....................................................................................................... 37
3.2.2 Metrics .................................................................................................................................. 37
3.2.3 Screen Video Capture........................................................................................................... 37
3.2.4 Observational Capture.......................................................................................................... 37
3.2.5 Configuration Support...........................................................................................................38
3.2.6 Event definitions ................................................................................................................... 38
4 Desirable Tool Features...................................................................................................... 40
4.1 Abstract Descriptions ..................................................................................................... 40
4.1.1 Usability Problem Instance Records ..................................................................................... 40
4.1.2 Diagnosis.............................................................................................................................. 41
4.1.3 Merging and Grouping .......................................................................................................... 42
4.2 Specific Instances .......................................................................................................... 42
4.2.1 Usability Problem Instance Records ..................................................................................... 43
4.2.2 Diagnosis.............................................................................................................................. 47
4.2.3 Merging and Grouping .......................................................................................................... 53

Frequently asked questions

Q1. What are the contributions mentioned in the paper "Supporting novice usability practitioners with usability engineering tools" ?

I developed a usability engineering tool, the Data Collection, Analysis, and Reporting Tool ( DCART ), which contains these desirable tool features, and used it as a platform for studies of how these desirable features address the documented difficulties. The results of the studies suggest that appropriate tool support can improve the effectiveness with which novice usability practitioners perform usability evaluations. Additionally, they show that appropriate tool support is a mechanism for further integrating usability engineering into the overall software development lifecycle ; tool support addresses the documented need for more usability practitioners by helping novices perform more like experts.