Usability evaluation methods for the web: A systematic mapping study
TL;DR: A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process, which provided a framework in which new research activities can be more appropriately positioned and from which useful information for novice usability practitioners can be extracted.
Abstract: Context: In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers' usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring. Objective: The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14years. Method: A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner. Results: The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners. Conclusions: From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.
Citations
More filters
TL;DR: There was a need to provide an update of how to conduct systematic mapping studies and how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines.
Abstract: Context Systematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines. Objective To identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly. Method We conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment). Results In a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given. Conclusion The most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings.
1,598 citations
Cites background or methods from "Usability evaluation methods for th..."
...(2008) [37,77,81] Dyb and Dingsyr (2008) [83] Bailey et al....
[...]
...Identify objective criteria for decision [80,20,45,37] Add additional reviewer, resolve disagreements between them when needed [73,64,76,41,6,30,80,20, 72,33,60,77,45,85,86,13,37]...
[...]
...Journal [72,33,12,34,74,35,36,13,76,37,64,22,20,54,38,80,30,81,21, 85,55,53,23] Conference [69,3,60,70,63,75,4,77,42,58,32,78,6,43,39,44,82–...
[...]
...Identify objective criteria/evaluate objectivity [46,36,76,37,22,54,85,35] Add additional reviewer, resolve disagreements between them when needed [39,53,69,23,4,30,73,77, 80,46,34,71,3,60,76,85,86,...
[...]
...Test-set of known papers [71,32,34,13,76,37,22,55] Expert evaluates result [39] Search web-pages of key authors [53]...
[...]
TL;DR: The objective is to understand the current research topics, challenges and future directions regarding Blockchain technology from the technical perspective, and recommendations on future research directions are provided for researchers.
Abstract: Blockchain is a decentralized transaction and data management technology developed first for Bitcoin cryptocurrency. The interest in Blockchain technology has been increasing since the idea was coined in 2008. The reason for the interest in Blockchain is its central attributes that provide security, anonymity and data integrity without any third party organization in control of the transactions, and therefore it creates interesting research areas, especially from the perspective of technical challenges and limitations. In this research, we have conducted a systematic mapping study with the goal of collecting all relevant research on Blockchain technology. Our objective is to understand the current research topics, challenges and future directions regarding Blockchain technology from the technical perspective. We have extracted 41 primary papers from scientific databases. The results show that focus in over 80% of the papers is on Bitcoin system and less than 20% deals with other Blockchain applications including e.g. smart contracts and licensing. The majority of research is focusing on revealing and improving limitations of Blockchain from privacy and security perspectives, but many of the proposed solutions lack concrete evaluation on their effectiveness. Many other Blockchain scalability related challenges including throughput and latency have been left unstudied. On the basis of this study, recommendations on future research directions are provided for researchers.
1,528 citations
TL;DR: The results show that the empirical evaluation methods employed as regards usability could be improved by the adoption of automated mechanisms, and the evaluation processes should also be revised to combine more than one method.
Abstract: The release of smartphones and tablets, which offer more advanced communication and computing capabilities, has led to the strong emergence of mHealth on the market. mHealth systems are being used to improve patients' lives and their health, in addition to facilitating communication between doctors and patients. Researchers are now proposing mHealth applications for many health conditions such as dementia, autism, dysarthria, Parkinson's disease, and so on. Usability becomes a key factor in the adoption of these applications, which are often used by people who have problems when using mobile devices and who have a limited experience of technology. The aim of this paper is to investigate the empirical usability evaluation processes described in a total of 22 selected studies related to mHealth applications by means of a Systematic Literature Review. Our results show that the empirical evaluation methods employed as regards usability could be improved by the adoption of automated mechanisms. The evaluation processes should also be revised to combine more than one method. This paper will help researchers and developers to create more usable applications. Our study demonstrates the importance of adapting health applications to users' need.
415 citations
04 Nov 2016
TL;DR: This paper presents a systematic mapping study of microservices architectures and their implementation, focusing on identifying architectural challenges, the architectural diagrams/views and quality attributes related to microsevice systems.
Abstract: The accelerating progress of network speed, reliability and security creates an increasing demand to move software and services from being stored and processed locally on users' machines to being managed by third parties that are accessible through the network. This has created the need to develop new software development methods and software architectural styles that meet these new demands. One such example in software architectural design is the recent emergence of the microservices architecture to address the maintenance and scalability demands of online service providers. As microservice architecture is a new research area, the need for a systematic mapping study is crucial in order to summarise the progress so far and identify the gaps and requirements for future studies. In this paper we present a systematic mapping study of microservices architectures and their implementation. Our study focuses on identifying architectural challenges, the architectural diagrams/views and quality attributes related to microsevice systems.
272 citations
Cites background from "Usability evaluation methods for th..."
...the guidelines outlined in [5], [33], [18] and [9]....
[...]
TL;DR: An IoT agriculture framework has been presented that contextualizes the representation of a wide range of current solutions in the field of agriculture and open issues and challenges have been presented to provide the researchers promising future directions in the domain of IoT agriculture.
Abstract: The growing demand for food in terms of quality and quantity has increased the need for industrialization and intensification in the agriculture field. Internet of Things (IoT) is a highly promising technology that is offering many innovative solutions to modernize the agriculture sector. Research institutions and scientific groups are continuously working to deliver solutions and products using IoT to address different domains of agriculture. This paper presents a systematic literature review (SLR) by conducting a survey of IoT technologies and their current utilization in different application domains of the agriculture sector. The underlying SLR has been compiled by reviewing research articles published in well-reputed venues between 2006 and 2019. A total of 67 papers were carefully selected through a systematic process and classified accordingly. The primary objective of this systematic study is the collection of all relevant research on IoT agricultural applications, sensors/devices, communication protocols, and network types. Furthermore, it also discusses the main issues and challenges that are being investigated in the field of agriculture. Moreover, an IoT agriculture framework has been presented that contextualizes the representation of a wide range of current solutions in the field of agriculture. Similarly, country policies for IoT-based agriculture have also been presented. Lastly, open issues and challenges have been presented to provide the researchers promising future directions in the domain of IoT agriculture.
179 citations
Cites methods from "Usability evaluation methods for th..."
...The QA in this SLR is carried out by following the previous mapping study [23]....
[...]
References
More filters
Book•
01 Jan 1981TL;DR: In this paper, the basic theory of Maximum Likelihood Estimation (MLE) is used to detect a difference between two different proportions of a given proportion in a single proportion.
Abstract: Preface.Preface to the Second Edition.Preface to the First Edition.1. An Introduction to Applied Probability.2. Statistical Inference for a Single Proportion.3. Assessing Significance in a Fourfold Table.4. Determining Sample Sizes Needed to Detect a Difference Between Two Proportions.5. How to Randomize.6. Comparative Studies: Cross-Sectional, Naturalistic, or Multinomial Sampling.7. Comparative Studies: Prospective and Retrospective Sampling.8. Randomized Controlled Trials.9. The Comparison of Proportions from Several Independent Samples.10. Combining Evidence from Fourfold Tables.11. Logistic Regression.12. Poisson Regression.13. Analysis of Data from Matched Samples.14. Regression Models for Matched Samples.15. Analysis of Correlated Binary Data.16. Missing Data.17. Misclassification Errors: Effects, Control, and Adjustment.18. The Measurement of Interrater Agreement.19. The Standardization of Rates.Appendix A. Numerical Tables.Appendix B. The Basic Theory of Maximum Likelihood Estimation.Appendix C. Answers to Selected Problems.Author Index.Subject Index.
16,435 citations
9,528 citations
Additional excerpts
...7 << Insert Table 1 approximately here >>...
[...]
Book•
01 Feb 1997TL;DR: The human and the design of interactive systems: The myth of the infinitely fast machine, a guide to designing for diversity and the process of design.
Abstract: Contents Foreword Preface to the third edition Preface to the second edition Preface to the first edition Introduction Part 1 Foundations Chapter 1 The human 1.1 Introduction 1.2 Input-output channels Design Focus: Getting noticed Design Focus: Where's the middle? 1.3 Human memory Design Focus: Cashing in Design Focus: 7 +- 2 revisited 1.4 Thinking: reasoning and problem solving Design Focus: Human error and false memories 1.5 Emotion 1.6 Individual differences 1.7 Psychology and the design of interactive systems 1.8 Summary Exercises Recommended reading Chapter 2 The computer 2.1 Introduction Design Focus: Numeric keypads 2.2 Text entry devices 2.3 Positioning, pointing and drawing 2.4 Display devices Design Focus: Hermes: a situated display 2.5 Devices for virtual reality and 3D interaction 2.6 Physical controls, sensors and special devices Design Focus: Feeling the road Design Focus: Smart-Its - making sensors easy 2.7 Paper: printing and scanning Design Focus: Readability of text 2.8 Memory 2.9 Processing and networks Design Focus: The myth of the infinitely fast machine 2.10 Summary Exercises Recommended reading Chapter 3 The interaction 3.1 Introduction 3.2 Models of interaction Design Focus: Video recorder 3.3 Frameworks and HCI 3.4 Ergonomics Design Focus: Industrial interfaces 3.5 Interaction styles Design Focus: Navigation in 3D and 2D 3.6 Elements of the WIMP interface Design Focus: Learning toolbars 3.7 Interactivity 3.8 The context of the interaction Design Focus: Half the picture? 3.9 Experience, engagement and fun 3.10 Summary Exercises Recommended reading Chapter 4 Paradigms 4.1 Introduction 4.2 Paradigms for interaction 4.3 Summary Exercises Recommended reading Part 2 Design process Chapter 5 Interaction design basics 5.1 Introduction 5.2 What is design? 5.3 The process of design 5.4 User focus Design Focus: Cultural probes 5.5 Scenarios 5.6 Navigation design Design Focus: Beware the big button trap Design Focus: Modes 5.7 Screen design and layout Design Focus: Alignment and layout matter Design Focus: Checking screen colors 5.8 Iteration and prototyping 5.9 Summary Exercises Recommended reading Chapter 6 HCI in the software process 6.1 Introduction 6.2 The software life cycle 6.3 Usability engineering 6.4 Iterative design and prototyping Design Focus: Prototyping in practice 6.5 Design rationale 6.6 Summary Exercises Recommended reading Chapter 7 Design rules 7.1 Introduction 7.2 Principles to support usability 7.3 Standards 7.4 Guidelines 7.5 Golden rules and heuristics 7.6 HCI patterns 7.7 Summary Exercises Recommended reading Chapter 8 Implementation support 8.1 Introduction 8.2 Elements of windowing systems 8.3 Programming the application Design Focus: Going with the grain 8.4 Using toolkits Design Focus: Java and AWT 8.5 User interface management systems 8.6 Summary Exercises Recommended reading Chapter 9 Evaluation techniques 9.1 What is evaluation? 9.2 Goals of evaluation 9.3 Evaluation through expert analysis 9.4 Evaluation through user participation 9.5 Choosing an evaluation method 9.6 Summary Exercises Recommended reading Chapter 10 Universal design 10.1 Introduction 10.2 Universal design principles 10.3 Multi-modal interaction Design Focus: Designing websites for screen readers Design Focus: Choosing the right kind of speech Design Focus: Apple Newton 10.4 Designing for diversity Design Focus: Mathematics for the blind 10.5 Summary Exercises Recommended reading Chapter 11 User support 11.1 Introduction 11.2 Requirements of user support 11.3 Approaches to user support 11.4 Adaptive help systems Design Focus: It's good to talk - help from real people 11.5 Designing user support systems 11.6 Summary Exercises Recommended reading Part 3 Models and theories Chapter 12 Cognitive models 12.1 Introduction 12.2 Goal and task hierarchies Design Focus: GOMS saves money 12.3 Linguistic models 12.4 The challenge of display-based systems 12.5 Physical and device models 12.6 Cognitive architectures 12.7 Summary Exercises Recommended reading Chapter 13 Socio-organizational issues and stakeholder requirements 13.1 Introduction 13.2 Organizational issues Design Focus: Implementing workflow in Lotus Notes 13.3 Capturing requirements Design Focus: Tomorrow's hospital - using participatory design 13.4 Summary Exercises Recommended reading Chapter 14 Communication and collaboration models 14.1 Introduction 14.2 Face-to-face communication Design Focus: Looking real - Avatar Conference 14.3 Conversation 14.4 Text-based communication 14.5 Group working 14.6 Summary Exercises Recommended reading Chapter 15 Task analysis 15.1 Introduction 15.2 Differences between task analysis and other techniques 15.3 Task decomposition 15.4 Knowledge-based analysis 15.5 Entity-relationship-based techniques 15.6 Sources of information and data collection 15.7 Uses of task analysis 15.8 Summary Exercises Recommended reading Chapter 16 Dialog notations and design 16.1 What is dialog? 16.2 Dialog design notations 16.3 Diagrammatic notations Design Focus: Using STNs in prototyping Design Focus: Digital watch - documentation and analysis 16.4 Textual dialog notations 16.5 Dialog semantics 16.6 Dialog analysis and design 16.7 Summary Exercises Recommended reading Chapter 17 Models of the system 17.1 Introduction 17.2 Standard formalisms 17.3 Interaction models 17.4 Continuous behavior 17.5 Summary Exercises Recommended reading Chapter 18 Modeling rich interaction 18.1 Introduction 18.2 Status-event analysis 18.3 Rich contexts 18.4 Low intention and sensor-based interaction Design Focus: Designing a car courtesy light 18.5 Summary Exercises Recommended reading Part 4 Outside the box Chapter 19 Groupware 19.1 Introduction 19.2 Groupware systems 19.3 Computer-mediated communication Design Focus: SMS in action 19.4 Meeting and decision support systems 19.5 Shared applications and artifacts 19.6 Frameworks for groupware Design Focus: TOWER - workspace awareness Exercises Recommended reading Chapter 20 Ubiquitous computing and augmented realities 20.1 Introduction 20.2 Ubiquitous computing applications research Design Focus: Ambient Wood - augmenting the physical Design Focus: Classroom 2000/eClass - deploying and evaluating ubicomp 20.3 Virtual and augmented reality Design Focus: Shared experience Design Focus: Applications of augmented reality 20.4 Information and data visualization Design Focus: Getting the size right 20.5 Summary Exercises Recommended reading Chapter 21 Hypertext, multimedia and the world wide web 21.1 Introduction 21.2 Understanding hypertext 21.3 Finding things 21.4 Web technology and issues 21.5 Static web content 21.6 Dynamic web content 21.7 Summary Exercises Recommended reading References Index
5,095 citations
28 May 2006
TL;DR: This tutorial is designed to provide an introduction to the role, form and processes involved in performing Systematic Literature Reviews, and to gain the knowledge needed to conduct systematic reviews of their own.
Abstract: Context: Making best use of the growing number of empirical studies in Software Engineering, for making decisions and formulating research questions, requires the ability to construct an objective summary of available research evidence. Adopting a systematic approach to assessing and aggregating the outcomes from a set of empirical studies is also particularly important in Software Engineering, given that such studies may employ very different experimental forms and be undertaken in very different experimental contexts.Objectives: To provide an introduction to the role, form and processes involved in performing Systematic Literature Reviews. After the tutorial, participants should be able to read and use such reviews, and have gained the knowledge needed to conduct systematic reviews of their own.Method: We will use a blend of information presentation (including some experiences of the problems that can arise in the Software Engineering domain), and also of interactive working, using review material prepared in advance.
4,352 citations
"Usability evaluation methods for th..." refers background in this paper
...Although several studies concerning UEMs have been reported, we are not aware of any systematic mapping study that has been published in the field of Web usability apart from our preliminary systematic mapping study (Insfran and Fernandez [18])....
[...]