Showing papers on "Acceptance testing published in 2019"

Smart City Bus Application with Quick Response (QR) Code Payment

Abstract: Mobile application that is available in Android operating system can do many functions such as access to the internet wirelessly, take digital photos, or even locate itself using Global Positioning System. Smart City Bus Application is an application that assist users in providing public buses' information such as bus route view on map, quick response (QR) code payment, etc. This paper however will discuss about the results from system testing and passengers' satisfaction view of each of the system's functions. System testing was performed using black-box method by sorting to project's objectives. The result from system testing is all positive meaning it is performed as expected from given input. In the other hand, acceptance testing was conducted by giving out the application to get evaluated by selected people that uses bus frequently. After testing, questionnaire where given to score each system's feature scaling from 1 to 5 as "Not Satisfy" and "Very Satisfy" respectively. In the end of the questionnaire, system's overall score was asked and the result is mostly positive.

Behavior-driven Load Testing Using Contextual Knowledge - Approach and Experiences

Abstract: Load testing is widely considered a meaningful technique for performance quality assurance. However, empirical studies reveal that in practice, load testing is not applied systematically, due to the sound expert knowledge required to specify, implement, and execute load tests.Our Behavior-driven Load Testing (BDLT) approach eases load test specification and execution for users with no or little expert knowledge. It allows a user to describe a load test in a template-based natural language and to rely on an automated framework to execute the test. Utilizing the system's contextual knowledge such as workload-influencing events, the framework automatically determines the workload and test configuration. We investigated the applicability of our approach in an industrial case study, where we were able to express four load test concerns using BDLT and received positive feedback from our industrial partner. They understood the BDLT definitions well and proposed further applications, such as the usage for software quality acceptance criteria.

Application of TG-100 risk analysis methods to the acceptance testing and commissioning process of a Halcyon linear accelerator.

Abstract: Purpose A new type of linear accelerator (linac) was recently introduced into the market by a major manufacturer. Our institution is one of the early users of this preassembled and preconfigured dual-layer multileaf collimator (MLC), ring-gantry linac - Halcyon™ (1st version). We performed a set of full acceptance testing and commissioning (ATC) measurements for three Halcyon machines and compared the measured data with the standard beam model provided by the manufacturer. The ATC measurements were performed following the guidelines given in different AAPM protocols as well as guidelines provided by the manufacturer. The purpose of the present work was to perform a risk assessment of the ATC process for this new type of linac and investigate whether the results obtained from this analysis could potentially be used as a guideline for improving the design features of this type of linac. Methods AAPM's TG100 risk assessment methodology was applied to the ATC process. The acceptance testing process relied heavily on the use of a manufacturer-supplied phantom and the automated analysis of electronic portal imaging device (EPID) images. For the commissioning process, a conventional measurement setup and process (e.g., use of water tank for scanning) was largely used. ATC was performed using guidelines recommended in various AAPM protocols (e.g., TG-106, TG-51) as well as guidelines provided by the manufacturer. Six medical physicists were involved in this study. Process maps, process steps, and failure modes (FMs) were generated for the ATC procedures. Failure modes and effects analysis (FMEA) were performed following the guidelines given in AAPM TG-100 protocol. The top 5 and top 10 highest-ranked FMs were identified for the acceptance and commissioning procedures, respectively. Quality control measures were suggested to mitigate these FMs. Results A total of 38 steps and 88 FMs were identified for the entire ATC process. Fourteen steps and 34 FMs arose from acceptance testing. The top 5 FMs that were identified could potentially be mitigated by the manufacturer. For commissioning, a total of 24 steps and 54 potential FMs were identified. The use of separate measurement tools that are not machine-integrated has been identified as a cause for the higher number of steps and FMs generated from the conventional ATC approach. More than half of the quality control measures recommended for both acceptance and commissioning could potentially be incorporated by the manufacturer in the design of the Halcyon machine. Conclusion This paper presents the results of FMEA and quality control measures to mitigate the FMs for the ATC process for Halcyon machine. Unique FMs that result from the differences in the ATC guidelines provided by the vendor and current conventional protocols, and the challenges of performing the ATC due to the new linac features and ring-gantry design were highlighted for the first time. The FMs identified in the present work along with the suggested quality control measures, could potentially be used to improve the design features of future ring-gantry type of linacs that are likely to be preassembled, preconfigured, and heavily reliant on automation and image guidance.

Usability Testing based on System Usability Scale and Net Promoter Score

Abstract: In a variety of software development methods and their models of the activity process, no one does not have testing activities in their application. System testing is the last phase of testing that done by the developer in the internal work- space. However, phase of testing is not the end of the whole testing phase, since there is another stage to pass before the product deployment process, it is the acceptance testing. A popular method to perform the user acceptance testing is the usability testing, which in this method, the result provides the indication of the essential quality for the client to decide whether to approve or to deny the product. In this research of paper, we are going to do the usability testing by utilizing the System Usability Scale technique and the Net Promoter Score technique to measure appropriateness of the population information systems in the city of Tegal while we are also finding optimum testers required using the sampling technique. The result we earn trough this research is according to the Usability Testing result the proposed system developed by researchers is better than the current running system and the total testers participating on system testing is able to optimize using the sampling method.

Towards continuous software reliability testing in DevOps

Abstract: We introduce the DevOpRET approach for continuous reliability testing in DevOps. It leverages information monitored in operation to guide operational-profile based testing, which is conceived as part of the acceptance testing stage before each next release to production. We overview the envisaged test and monitoring pipeline, describe the approach and present a case-study evaluating how reliability assessment evolves over subsequent releases.

Formal Model Validation Through Acceptance Tests

Abstract: When formal systems modelling is used as part of the development process, modellers need to understand the requirements in order to create appropriate models, and domain experts need to validate the final models to ensure they fit the needs of stakeholders. A suitable mechanism for such a validation are acceptance tests.

On the Evolution of Keyword-Driven Test Suites

Abstract: Many companies rely on software testing to verify that their software products meet their requirements. However, test quality and, in particular, the quality of end-to-end testing is relatively hard to achieve. The problem becomes challenging when software evolves, as end-to-end test suites need to adapt and conform to the evolved software. Unfortunately, end-to-end tests are particularly fragile as any change in the application interface, e.g., application flow, location or name of graphical user interface elements, necessitates a change in the tests. This paper presents an industrial case study on the evolution of Keyword-Driven test suites, also known as Keyword-Driven Testing (KDT). Our aim is to demonstrate the problem of test maintenance, identify the benefits of Keyword-Driven Testing and overall improve the understanding of test code evolution (at the acceptance testing level). This information will support the development of automatic techniques, such as test refactoring and repair, and will motivate future research. To this end, we identify, collect and analyze test code changes across the evolution of industrial KDT test suites for a period of eight months. We show that the problem of test maintenance is largely due to test fragility (most commonly-performed changes are due to locator and synchronization issues) and test clones (over 30% of keywords are duplicated). We also show that the better test design of KDT test suites has the potential for drastically reducing (approximately 70%) the number of test code changes required to support software evolution. To further validate our results, we interview testers from BGL BNP Paribas and report their perceptions on the advantages and challenges of keyword-driven testing.

From Requirements to Automated Acceptance Tests with the RSL Language

Abstract: Software testing can promote software quality. However, this activity is often performed at the end of projects where failures are most difficult to correct. Combining requirements specification activities with test design at an early stage of the software development process can be beneficial. One way to do this is to use a more structured requirements specification language. This allow to reduce typical problems such as ambiguity, inconsistency, and incorrectness in requirements and may allow the automatic generation of (parts of) acceptance test cases reducing the test design effort. In this paper we discuss an approach that promotes the practice of requirements specification combined with testing specification. This is a model-based approach that promotes the alignment between requirements and tests, namely, test cases and also low-level automated test scripts. To show the applicability of this approach, we integrate two complementary languages: (i) the ITLingo RSL (Requirements Specification Language) that is specially designed to support both requirements and tests rigorously and consistently specified; and (ii) the Robot language, which is a low-level keyword-based language for specifying test scripts. This approach includes model-to-model transformation processes, namely a transformation process from requirements (defined in RSL) into test cases (defined in RSL), and a second transformation process from test cases (in RSL) into test scripts (defined according the Robot framework). This approach was applied in a fictitious online store that illustrates the various phases of the proposal.

From Requirements to Automated Acceptance Tests of Interactive Apps: An Integrated Model-based Testing Approach

Abstract: Frequently software testing tends to be neglected at the beginning of the projects, only performed on the late stage. However, it is possible to benefit from combining requirement with testing specification activities. On one hand, acceptance tests specification will require less manual effort since they are defined or generated automatically from the requirements specification. On the other hand, the requirements specification itself will end up having higher quality due to the use of a more structured language, reducing typical problems such as ambiguity, inconsistency and incorrectness. This research proposes an approach that promotes the practice of tests specification since the very beginning of projects, and its integration with the requirements specification itself. It is a model-driven approach that contributes to maintain the requirements and tests alignment, namely between requirements, test cases, and low-level automated test scripts. To show the applicability of the approach, two complementary languages are adopted: the ITLingo RSL that is particularly designed to support both requirements and tests specification; and the Robot language, which is a low-level keyword-based language for the specification of test scripts. The approach includes model-to-model transformation techniques, such as test cases into test scripts transformations. In addition, these test scripts are executed by the Robot test automation framework.

Enhancing Acceptance Test-Driven Development with Model-Based Test Generation

Abstract: Acceptance test-driven development is widely used in practice. However, writing and maintaining acceptance tests is a costly and time-consuming activity, in particular when a system is tested via the GUI. In model-based testing, the tests are automatically generated from a model of the system. In this paper, we report our experience from applying a combination of acceptance test-driven development and model-based testing in several real-world projects from industry. With the application of model-based testing, we increased test coverage and extend testing to usage scenarios not exercised by the existing acceptance tests. In the industry projects, MBT was used as an enhancement rather than a replacement for ATDD. By creating a layered test automation architecture, we were able to reuse the established automation for model-based testing and to apply both approaches simultaneously. This strategy also helped us to minimize the risks and to reduce the effort involved in introducing MBT in the projects.

GuideGen: An approach for keeping requirements and acceptance tests aligned via automatically generated guidance

Abstract: Context When software-based systems evolve, their requirements change. The changes in requirements affect the associated acceptance tests, which should be adapted accordingly. In practice, however, requirements and their acceptance tests are not always kept up-to-date nor aligned. Such inconsistencies may introduce software quality problems, unintended costs and project delays. Objective In order to keep evolving requirements and their acceptance tests aligned, we are developing an approach called GuideGen. GuideGen automatically generates guidance in natural language about how to adapt the impacted acceptance tests when their requirements change. Method We have implemented GuideGen as a prototype tool and evaluated it in two studies: first, by assessing the correctness, completeness, understandability and relevance of the generated guidance using three data sets from industry and second, by assessing the applicability and usefulness of the approach and the tool with 23 practitioners from ten companies. When a requirement having more than one associated acceptance test is changed, GuideGen currently generates guidance for all of them together. As a first step towards overcoming this limitation, we assessed how well existing methods for change impact analysis can identify the tests actually impacted by the changes in a requirement. Results In the first study, we found that GuideGen produced correct guidance in about 67 to 89 percent of all changes. Our approach performed better for agile requirements than for traditional ones. The results of the second study show that GuideGen is perceived to be useful, but that the practitioners would prefer a GuideGen plug-in for commercial tools instead of a standalone tool. Further, in our experiment we could correctly identify the affected acceptance tests for 63% to 91% of the changes in the requirements. Conclusion Our approach facilitates the alignment of acceptance tests with the actual requirements and can improve the communication between requirements engineers and testers.

A compatibility testing platform for android multimedia applications

Abstract: Along with the widespread use of smartphones, Android has become one of the major platforms for multimedia applications (apps). However, due to the fast evolution of Android operating system and the fragmentation of Android devices, it becomes important for an Android multimedia app to be tested on different devices to ensure that the app is compatible with and run well on any of the devices so as to provide consistent user experiences. This paper presents a cloud testing platform (CTP) that allows Android multimedia apps to be tested automatically against a scalable number of physical devices in parallel. Particularly, CTP provides five types of testing to ensure the compatibility of apps from different perspectives, including GUI testing, acceptance testing, stress testing, crash testing, and installation/uninstallation testing. Further, to facilitate identifying the bugs of apps, in addition to test results, CTP also provides the video, screenshots, and performance data corresponding to the tests. Moreover, CTP can also generate a GUI state diagram that can be used to analyze app’s behavior and is helpful for crash diagnosis and debugging. The case study shows that CTP can be effective in ensuring the compatibility of Android multimedia apps while saving test time and effort.

Comparing Testing and Runtime Verification of IoT Systems: A Preliminary Evaluation based on a Case Study

Abstract: Assuring the quality of Internet of Things (IoT) systems is of paramount importance, and guaranteeing their reliability and compliance with the requirements is mandatory, but few attempts have been made so far. In previous works, we proposed two approaches for acceptance testing and runtime verification of IoT systems. Both works rely on a UML state machine to specify the system expected behaviour. In the acceptance testing approach, the interesting paths to exercise are identified and translated into executable test scripts. In the runtime verification approach, the relevant events during the system execution are monitored and compared against a formal specification derived from the UML state machine. In this paper, we compare the effectiveness of our two approaches, by applying them to a mobile health IoT system for the management of diabetic patients, employing over 100 mutated versions of the original system and analysing more than 1000 different executions. Results show that both approaches are effective in different ways in detecting bugs. While the acceptance testing approach is more effective to detect the bugs affecting the user interface, the runtime verification approach tracks better the subtle deviations from the system expected behaviour, in particular those concerning network issues.

Modelling drivers of Koha open source library system using partial least squares structural equation modelling

Abstract: Open source library systems are markedly a less expensive alternative to licensed library systems, but the uptake by libraries in the developing countries has been slower than expected. Focusing on the user, this study employs a human behavior analysis approach to ascertain what factors drive librarians to accept an open source library information system, specifically the Koha library system. This exploratory study is guided by two important components of technology acceptance, the Unified Theory of Acceptance and Use of Technology and the User Acceptance Test. Using a survey questionnaire, data was collected from 245 librarians from five public and private university libraries and analyzed using the PLS-SEM method. Librarians’ attitude towards using technology and social influence are the key determinants of open source library system acceptance. At the same time, system quality and information quality are also two other significant predictors of librarians’ acceptance of open source technology. This study is the first to conceptually integrate technology acceptance predictors and open system success indicators, as used in user acceptance tests, to understand open source library system acceptance and use among university librarians. The identified major predictors can be used by library management to devise strategies to improve librarians’ involvement and use of open source systems.

User Acceptance Testing in Mobile Health Applications: An overview and the Challenges

Abstract: The software development trend has moved from the stand alone applications to the development of the mobile applications that had widely used nowadays. The software testing also reflected with this changes as it is needed to suit with this movement. Today, the mobile health applications had getting great attention from all people around the world and become the most important tools for the users. In order to promote confidence in using the mobile health applications, an acceptance testing needs to be conducted. Due to the importance and criticality of health, there is an urgency of needs to conduct the research in this particular area. Thus, the research is conducted to identify the challenges in conducting the UAT in M-Health applications.

Testing and validation of safety logic in the virtual environment

Abstract: This paper presents an approach for testing safety PLC logic in a virtual environment, using the IOCO testing relation as validation criteria. Manufacturing companies more and more rely on virtual commissioning to reduce the physical commissioning time by testing and debugging the PLC logic of the nominal behavior prior to physical commissioning. However, safety PLC logic testing is still carried out on real systems manually. This manual practice of safety logic validation hinders industry to exploit the full potential of virtual commissioning to reduce the physical commissioning time. The proposed approach assists manufacturing companies in the validation of safety PLC logic using a simulation model before the factory acceptance testing phase. Using the proposed approach, a simulation model can be used to test the safety PLC logic and prepare better for the factory acceptance testing phase, hence, further reduction in physical commissioning time can be achieved.

Eu:CROPIS AIV Program: Challenges and Solutions for a Spin-Stabilized Satellite Containing Biology

Abstract: Eu:CROPIS is DLR’s first mission of the Compact Satellite Program. Its primary payload focuses on the research of closed-loop biological, regenerative life support systems, in a simulated gravitational environment of the Moon and Mars over months at a time. This is achieved by rotation of the satellite around its central body axis, using only magnetic torquers as actuators. A secondary payload (“PowerCells”) by the NASA Ames Research Center also utilizes the artificial gravity to conduct growth experiments on genetically modified organisms (GMOs). These payloads and the system design imposed constraints which affected the Assembly Integration and Verification (AIV) program in various ways and created challenges for the relatively small team to find solutions for. The paper to be presented will address the different aspects of the AIV program. This includes the verification of different critical components like the newly developed CFRP pressure vessel containing the primary payload and the Micrometeoroid and Debris Protection Shield, which protects it. Both items went through rigorous testing, including high-velocity impact tests, to ensure their reliability in orbit. Various other aspects concerning the biology had to be taken into account during AIV campaigns: due to the presence of degradable components within the primary payload, a late access capability had to be implemented in order to exchange biology as well as chemistry in cases of launch delays. To allow these operations as close as six months prior to launch, a highly flexible and streamlined acceptance test campaign was developed. A major impact on test planning and logistics was the fact that the secondary payload “PowerCells” contains GMOs, which European and German regulations restrict to be handled exclusively in especially certified laboratories (biosafety level 1 (BSL-1)). Thus, the use of external test facilities for the flight model campaign was not feasible as no European test center is certified to BSL-1. In consequence, the clean room facilities of the DLR Institute of Space Systems had to be certified to BSL-1 and new test infrastructure had to be procured in a short time frame to cover for acceptance testing. The design of the satellite and nature of the attitude control subsystem required limits on the magnetic momentum of the system and every unit it contains. A test flow incorporating the magnetic property measurement of each unit and a final system-level test in an external facility had to be devised, which enabled budgeting and projection of expected measurement results on the system level. Furthermore, the moments of inertia had to be measured precisely in order to have a stable spinning axis enabling a stable gravity simulation. Finally, the functionality had to be verified for each unit and for the system which required that several small test campaigns had to be conducted, like a solar panel deployment test and extensive software testing. A tight link to the operations teams of the German Space Operations and Control Center during such tests and beyond finally ensures the operability of the overall system in the operational phase.

Ukwikora: continuous inspection for keyword-driven testing

Abstract: Automation of acceptance test suites becomes necessary in the context of agile software development practices, which require rapid feedback on the quality of code changes. To this end, companies try to automate their acceptance tests as much as possible. Unfortunately, the growth of the automated test suites, by several automation testers, gives rise to potential test smells, i.e., poorly designed test code, being introduced in the test code base, which in turn may increase the cost of maintaining the code and creating new one. In this paper, we investigate this problem in the context of our industrial partner, BGL BNP Paribas, and introduce Ukwikora, an automated tool that statically analyzes acceptance test suites, enabling the continuous inspection of the test code base. Ukwikora targets code written in the Robot Framework syntax, a popular framework for writing Keyword-Driven tests. Ukwikora has been successfully deployed at BGL BNP Paribas, detecting issues otherwise unknown to the automation testers, such as the presence of duplicated test code, dead test code and dependency issues among the tests. The success of our case study reinforces the need for additional research and tooling for acceptance test suites.

Improvements on Thermal Performance of Power Transformers: Modelling and Testing

Abstract: Thermal performance of oil-immersed power transformer requires comprehensive solutions for the complex conjugate thermal-hydraulic problem which determines the thermal behavior of the oil and winding set. However, development of a thermal model without consideration of test procedures and limitations can led to inaccurate results during factory acceptance test (FAT). In the current study, we present improvements in modelling and testing based on an effective data exchange between both. Accurate loss calculation in transformer components and better investigation of oil behavior as well as development of more reliable rules to analyze heat-run results with consideration of new effective factors lead to a more accurate calculation of the thermal characteristics. This will increase the level of certainty in meeting the specified temperature rises during FAT, and therefore, accurate prediction of the transformer lifetime. Furthermore, minimum deviation between calculated and measured temperatures will allow optimization of the cooling system resulting in a more economical transformer design.

User acceptance testing (UAT) for the development and evaluation of an automated learning style detection system

Abstract: The purpose of this study is to determine the acceptable level of a proposed automated learning style detection system. This system was tested among teachers in schools based on a construct presented by the Unified Theory of Acceptance and Use of Technology (UTAUT) Model. A prototype application named VAKLeS was developed in order to evaluate user acceptance among the respondents. A combination of NeatBeans, Java, and MySQL was used to implement the learning style-based application to present the appropriate subject matter through content or activity, format, and platform type. VAKLeS was organized into two modules: the learning style module (student module) and decision support module (teacher module). The teacher acts as the decision maker while the student is the system user. The learning style of the user is evaluated based on his or her behavior when using the application. The two modules interact together to predict the user learning style. User testing was conducted for teachers to evaluate the performance of the system in detecting user learning style. VAKLeS was evaluated based on user acceptance testing (UAT). A set of questionnaires was distributed to ten respondents, who are domain expert in pre-school teaching. The questionnaire has 26 variables consisting of four core evaluation elements: performance expectations (nine variables), effort expectancy (seven variables), social influence (six variables), and facilitating conditions (three variables). These items are related to the usage of VAKLeS and are rated based on a seven-point Likert Scale. The finding of user acceptance was based on the average score of Strongly disagree (1) and Strongly agree (7). The findings of VAKLeS testing showed that the four elements evaluated was positive significantly with 5.87 for performance expectancy, 5.93 for effort expectancy, 6.03 for social influence, and 5.90 for facilitating condition. The finding of the UAT among the domain expert indicates that the proposed VAKLeS is acceptable and can be applied as an automated system for detecting VAK learning style based on the student’s behavioral pattern. Furthermore, the finding of this study also reveals that the proposed system, VAKLeS, significantly improved the ability of teachers to detect the learning style of their students.

Improving vibration test methods and profile selection for complex land vehicle payloads

Abstract: Mechanical vibration testing is used within the Department of Defence for acceptance testing of land materiel. Historical testing has commonly implemented default vibration profiles listed in defen...