One of the ways to evaluate a system is from Textual Evaluation. This type of evaluation consider the textual user’s opinions to infer aspects of the interaction with the system. Although this method covers many texts and consi- ders spontaneous narratives of the users, it takes a lot of time and effort. Some authors have reported on the need to compare evaluations techniques in order to investigate their effectiveness in revealing issue or to supplement the results of a systems assessment. This study presents a comparative analysis between the textual evaluation and user testing. A case study was performed evaluating the usability and user experience of a health app. As a result, the techniques were analyzed based on aspects that involved describing the results, resources needed and description of problems and users.

/pdf/textual-evaluation-vs-user-testing-a-comparative-analysis-4133plc1e7.pdf

Textual evaluation vs. User testing: a comparative analysis

User reviews of mobile applications in online stores have been used as a source for numerous studies. Many focus on indicating new functional requirements and extracting problems regarding quality characteristics, such as Usability. One of the most known strategies to evaluate Usability is Nielsen’s heuristics. We believe that user reviews in online stores can indicate heuristics issues, thus supporting evaluations more focused on the user feedback. Therefore, this study aims to investigate the presence of usability heuristic issues through user reviews. To achieve this goal, we collected and analyzed 200 reviews, split into 528 statements, from 10 free and paid Android and iOS apps. Our results showed that user reviews have certain potential to indicate heuristic issues, suggesting new and challenging future research for the CHI community.

Can User Reviews Indicate Usability Heuristic Issues?

Context: Digital transformation is a reality; Services that used to be face-to-face are now very flexible. With this, the countries chosen by the e-government improve the relationship with the citizens. Problem: Citizen-user systems have their own characteristics and a plurality of development needs and development needs to offer services. Solution: One of the ways to check the quality of computer systems is the Human-Computer Interaction (HCI) estimates, which facilitate the identification and correction of problems that can cause wear and tear in the user’s interaction with these systems. IS theory: It was based on the theory of technology acceptance (to the extent that technology accepts), which are important for the use and quality of the application by citizens. Method: We used: Semiotic Inspection Method (communicability), Heuristic Evaluation and Cognitive Path (usability), and MALTU (usability based on user comments in the application store). In addition to their specificities, all of them do not require direct participation from the end user during the process. Summarize: The combined use of the application of methods and problems in the overview of a comparative analysis with a focus on evaluation in applications. Contributions and Impact in the IS area: find yourself in the evaluation of an application that offers a diversity of services and thousands of Ceará users. Considering the challenges in the area, it was concerned with the quality of the interface and interaction. Help in deciding which method is most suitable for evaluating government application.

Digital Government and User Experience: how the combination of evaluation methods can contribute to the improvement of m-gov applications

Invisibility in Ubiquitous Computing (UbiComp) and Internet of Things (IoT) systems refers to the merge of technology in the environment or the interaction workload decrease so that users can focus on their daily activities. It has been a fundamental attribute for these systems since the UbiComp advent and has received different names and definitions over time. However, there is a lack of retrospective studies investigating how these perspectives contributed to the current concept of Invisibility. Therefore, this paper aims to understand how Invisibility evolved as a quality-in-use attribute. We applied the snowballing method to search for literature evidence about the attributes and interaction evaluation approaches related to Invisibility. We identified eight related attributes: Intelligibility, Accuracy, Control, Customization, Non-obtrusiveness, Diffusion, Transparency, and Understandability. Also, we observed that the existing evaluation approaches consider only part of these attributes. Our findings suggest opportunities to improve the approaches and for future research.

Hidden things and implicit interaction: looking back on how invisibility evolved as a quality-in-use attribute

System evaluation are useful for pointing out positive and negative aspects about its use. An evaluation expresses the results (problems and successes) of the moment it was performed. However the problems encountered can be solved, but they can also get worse over time. In the same way, novice users can accept or adapt to problems. In order to make an analysis considering the evolution of the system and users' opinions over time, more than one evaluation is usually done to investigate whether the problems encountered persisted, whether users' satisfaction increased, or whether criticisms diminished. Repeating evaluations requires effort, as they must always be applied in the same way so as not to bias the results. This article aims to consider the changes that have occurred in a system over time based on the perceptions of users. A textual evaluation was made in an academic management system adopted by federal universities of Brazil and was developed an algorithm of system analysis over time. Some of the obtained results were: the satisfaction of the users, the functionalities with problems, and the main factors that affected the interaction.

Temporal analysis of posts related to use: case study of use of an academic management system

Ubiquitous systems are made up of sensors and smart devices that communicate with each other to achieve certain goals. They have specific characteristics, such as transparency, attention, calmness, mobility, and context-awareness. Given that the Human-Computer Interaction (HCI) evaluation methods have been proposed for traditional systems, these characteristics present challenges that may require their adaptation. In this paper, we conducted an exploratory study aimed to compare the suitability of Heuristic Evaluation and the MALTU Model to evaluate the Usability and User eXperience (UX) of ubiquitous systems. For this, we apply both methods for evaluating a healthcare system and perform a comparative analysis. From our findings, MALTU is indicated for UX evaluation, being a low-cost approach when compared to Heuristic Evaluation, which is indicated for usability evaluation. Both methods were able to identify the problems that emerge from ubiquity and can provide better results when applied in aggregate. Furthermore, we provide recommendations to apply both methods for the Usability and UX evaluation of ubiquitous systems.

Comparing heuristic evaluation and MALTU model in interaction evaluation of ubiquitous systems

O objetivo do presente artigo e avaliar a usabilidade e a experiencia do usuario do aplicativo ubiquo Hidrate Spark, desenvolvido para apoiar a ingestao diaria de agua. Para isso, um conjunto de metodos foi utilizado, a fim de enfrentar o desafio de avaliar a qualidade de uso de sistemas ubiquos. Os resultados mostram os problemas que podem prejudicar a interacao do usuario com o aplicativo.

/pdf/hidrate-spark-avaliando-um-sistema-ubiquo-para-motivar-a-e1pj48r56p.pdf

Marcos Randel Freitas Brito

Papers

Comparing heuristic evaluation and MALTU model in interaction evaluation of ubiquitous systems

Hidrate Spark: Avaliando um Sistema Ubíquo para Motivar a Ingestão de Água