Showing papers on "Acceptance testing published in 2014"

On strategies for testing software product lines: A systematic literature review

Abstract: Context: Testing plays an important role in the quality assurance process for software product line engineering. There are many opportunities for economies of scope and scale in the testing activities, but techniques that can take advantage of these opportunities are still needed. Objective: The objective of this study is to identify testing strategies that have the potential to achieve these economies, and to provide a synthesis of available research on SPL testing strategies, to be applied towards reaching higher defect detection rates and reduced quality assurance effort. Method: We performed a literature review of two hundred seventy-six studies published from the year 1998 up to the 1st semester of 2013. We used several filters to focus the review on the most relevant studies and we give detailed analyses of the core set of studies. Results: The analysis of the reported strategies comprised two fundamental aspects for software product line testing: the selection of products for testing, and the actual test of products. Our findings indicate that the literature offers a large number of techniques to cope with such aspects. However, there is a lack of reports on realistic industrial experiences, which limits the inferences that can be drawn. Conclusion: This study showed a number of leveraged strategies that can support both the selection of products, and the actual testing of products. Future research should also benefit from the problems and advantages identified in this study.

A taxonomy of risk-based testing

Abstract: Software testing has often to be done under severe pressure due to limited resources and a challenging time schedule facing the demand to assure the fulfillment of the software requirements. In addition, testing should unveil those software defects that harm the mission-critical functions of the software. Risk-based testing uses risk (re-)assessments to steer all phases of the test process to optimize testing efforts and limit risks of the software-based system. Due to its importance and high practical relevance, several risk-based testing approaches were proposed in academia and industry. This paper presents a taxonomy of risk-based testing providing a framework to understand, categorize, assess, and compare risk-based testing approaches to support their selection and tailoring for specific purposes. The taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top-level classes risk drivers, risk assessment, and risk-based test process. The taxonomy of risk-based testing has been developed by analyzing the work presented in available publications on risk-based testing. Afterwards, it has been applied to the work on risk-based testing presented in this special section of the International Journal on Software Tools for Technology Transfer.

Diagnostic load testing and assessment of existing bridges: examples of application

Abstract: Load testing method is a significant tool in the assessment of bridge safety. One type of load tests is diagnostic load testing, the aim of which is to establish a comparison between real bridge behaviour and analytical calculation. It can be used either as acceptance test of the structures or as an estimation tool for the load carrying capacity of the already existing structures that have been in service for some time. This article presents diagnostic load tests and three examples of their application to various bridge structures and emphasises their diagnostic potential for assessment. In the majority of cases of diagnostic load tests presented in the literature, the experimental results (deflections, strains, etc) are very close or lower than the predicted ones and for this reason, such tests are expected to validate the existing structure. However, in the cases presented here, it is shown how the experimental results differ considerably from the expected ones. Since the theoretical and the experimental results of all three examples did not match, it was highly recommended to keep the matter under close investigation. A deep insight into the bridges showed serviceability and safety concerns and in some cases a repair/strengthening was necessary.

Using a real world project in a software testing course

Abstract: Although testing often accounts for 50% of the budget of a typical software project, the subject of software testing is often overlooked in computing curriculum. Students often view testing as a boring and unnecessary task, and education is usually focused on building software, not ensuring its quality. Previous works have focused on either making the subject of testing more exciting for students or on a more potent lecture-based learning process. At the Department of Software Engineering at the Rochester Institute of Technology, recent efforts have been focused on the project component of our Software Testing course as an area of innovation. Rather than previous methods such as a tightly controlled and repetitive testbed, our students are allowed to choose a real-world, open source project to test throughout the term. With the instructor as both counsel and client, students are expected to deliver a test plan, a final report, and several class-wide presentations. This project has achieved significant student praise; qualitative and quantitative feedback demonstrates both increased satisfaction and fulfilled curricular requirements. Students enjoy the real-world aspect of the project and the ability to work with relevant applications and technologies. This paper outlines the project details and educational goals.

On the applicability of combinatorial testing to web application security testing: a case study

Abstract: Case studies for evaluating tools in security testing are powerful. Although they cannot achieve the scientific rigor of formal experiments, the results can provide sufficient information to help professionals judge if a specific technology being evaluated will benefit their organization. This paper reports on a case study done for evaluating and revisiting a recently introduced combinatorial testing methodology used for web application security purposes. It further reports on undertaken practical experiments thus strengthening the applicability of combinatorial testing to web application security testing.

Murphy Tools: Utilizing Extracted GUI Models for Industrial Software Testing

Abstract: One of the main challenges in adopting model-based testing (MBT) is the effort and expertise required to produce the formal models. For an existing system, there are various approaches to automate the process of creating the models. In this paper, we share our experiences from a long term industrial evaluation on automatically extracting models of graphical user interface (GUI) applications and utilizing the extracted models to automate and support GUI testing. While model extraction and GUI testing has been recently a popular research topic, most proposed approaches have limitations on what can be modeled and industry adoption has been lacking. We describe the process of using Murphy tools to extract GUI models and utilize these models to automate and support various testing activities. During the evaluation, test engineers of an industrial software company used Murphy tools to support their daily efforts in testing commercial software products during 1 year time period. The results from the evaluation were promising, significantly reducing time and effort required for GUI testing.

Improving Models for Model-Based Testing Based on Exploratory Testing

Abstract: Model-based testing facilitates automatic generation of test cases by means of models of the system under test. Correctness and completeness of these models determine the effectiveness of the generated test cases. Critical faults can be missed due to omissions in the models, which are primarily created manually. In practice, these faults are usually detected with exploratory testing performed manually by experienced test engineers. In this paper, we propose an approach for refining system models based on the experience and domain knowledge of these test engineers. Our toolset analyzes the execution traces that are recorded during exploratory testing activities and identifies the omissions in system models. The identified omissions guide the refinement of models to be able to generate more effective test cases. We applied our approach in the context of an industrial case study to improve the models for model-based testing of a Digital TV system. After applying our approach, three critical faults were detected. These faults were not detected by the initial set of test cases and they were also missed during the exploratory testing activities.

A Risk Assessment Framework for Software Testing

Abstract: In industry, testing has to be performed under severe pressure due to limited resources Risk-based testing which uses risks to guide the test process is applied to allocate resources and to reduce product risks Risk assessment, ie, risk identification, analysis and evaluation, determines the significance of the risk values assigned to tests and therefore the quality of the overall risk-based test process In this paper we provide a risk assessment model and its integration into an established test process This framework is derived on the basis of best practices extracted from published risk-based testing approaches and applied to an industrial test process

Performance evaluation of web based automation testing tools

Abstract: In today's 21st century era countless software applications are written as a web based application which runs in a web browsers. With new technologies and commercialization of I.T. sector, the web based system has undergoes frequent and rapid changes. Today Softwares are coded as a web based application, which help to access data from any part of the globe. Even the economic relevance of web based enhances the control and quality of software. The quality assurance of any system depends on its test. But to do manually testing in most of the cases is time consuming, expensive and hectic. For the better business purpose and to save time and money automation testing is required. There are variety of tools are available in the market for this. One of the best known tool is selenium suite which is a combination of different automation testing tool. In this paper we will discuss about the selenium suite. It provides testers with different framework for different test cases. The main objective of this paper is to find the best tool in selenium suite and then compare it with some other tool for same task. For this purpose, performance evaluation is done on the basis of some criteria.

Agile development cycle: Approach to design an effective Model Based Testing with Behaviour driven automation framework

Abstract: Model Based Testing (MBT) is a relatively an approach widely discussed in software testing It extends test automation from test design to test execution using automatic test generation from models The effective use of this approach requires new skills, technology and knowledge, such as test modelling skills, but also good programs and tool support This paper explores a design of a Behaviour driven test automation framework using MBT and how it can be effectively used during Agile Development The automation framework is experimented upon the integration of the Graphwalker, a Model Based Graphical User Interface test generator with behaviour driven development framework and Robot Framework

Model-Based Testing

Abstract: Software more and more pervades our everyday lives. Hence, we have high requirements towards the trustworthiness of the software. Software testing greatly contributes to the quality assurance of modern software systems. However, as today's software system get more and more complex and exist in many different variants, we need rigorous and systematic approaches towards software testing. In this tutorial, we, first, present model-based testing as an approach for systematic test case generation, test execution and test result evaluation for single system testing. The central idea of model-based testing is to base all testing activities on an executable model-based test specification. Second, we consider model-based testing for variant-rich software systems and review two model-based software product line testing techniques. Sample-based testing generates a set of representative variants for testing, and variability-aware product line testing uses a family-based test model which contains the model-based specification of all considered product variants.

Integrating virtual and physical testing to accelerate the engineering product development process

Abstract: Testing is essential in developing a successful complex engineering product. System level integration and testing can use between 35% and 50% of development resources. External factors such as legislation and customer requirements drive essential testing whilst internal factors such company experience, affordability and organisational practice profoundly affect the overall testing plan. The main objective of this paper is to understand how testing is integrated into the product development process and how different types of testing are scheduled across the stages of product development. The paper reports a case study in a diesel engine company where the balance of virtual and physical testing is a key concern in reducing design time and cost. Integrating physical and virtual testing is more than process optimisation of time and cost. It contributes to recasting the design process in response to changes in customer requirements as well as to design changes which arise during testing. The importance of dependencies across components, subsystems and tests is highlighted using a model using design structure matrices and the advantages of integrating physical and virtual testing are analysed particularly in facilitating task overlap to reduce product development duration.

Supporting the Combined Selection of Model-Based Testing Techniques

Abstract: The technical literature on model-based testing (MBT) offers us several techniques with different characteristics and goals. Contemporary software projects usually need to make use of different software testing techniques. However, a lack of empirical information regarding their scalability and effectiveness is observed. It makes their application difficult in real projects, increasing the technical difficulties to combine two or more MBT techniques for the same software project. In addition, current software testing selection approaches offer limited support for the combined selection of techniques. Therefore, this paper describes the conception and evaluation of an approach aimed at supporting the combined selection of MBT techniques for software projects. It consists of an evidence-based body of knowledge with 219 MBT techniques and their corresponding characteristics and a selection process that provides indicators on the level of adequacy (impact indicator) amongst MBT techniques and software projects characteristics. Results from the data analysis indicate it contributes to improve the effectiveness and efficiency of the selection process when compared to another selection approach available in the technical literature. Aiming at facilitating its use, a computerized infrastructure, evaluated into an industrial context and evolved to implement all the facilities needed to support such selection approach, is presented.

Using artificial intelligence to automatically test GUI

Abstract: This position paper presents a synopsis of the imperative function artificial intelligence (AI) has partaken in software engineering (SE) as well as in software testing. In addition, the paper discusses how graphical user interface (GUI), and event driven software testing can derive benefits from the use of AI techniques. Artificial intelligence has significantly aided the process of the automation of different software process. The employment of AI in software testing is not novel, having played a crucial role in the automation of software testing since its innovation. The usage of AI techniques not only reduces the cost but it also guarantees better quality as well as thorough testing. GUI Testing can be considered as the most challenging area of software testing. Although the results are quite preliminary, but the application of different AI techniques for GUI testing has proven to produce ideal results. Nevertheless, the application of AI techniques in GUI testing, in comparison with software testing, which has procured much assistance by venturing with AI, leaves much to be desired.

A successful application of a Test-Driven Development strategy in the industrial environment

Abstract: Unit Test-Driven Development (UTDD) and Acceptance Test-Driven Development (ATDD) are software development techniques to incrementally develop software where the test cases, unit or acceptance tests respectively, are specified before the functional code. There are little empirical evidences supporting or refuting the utility of these techniques in an industrial context. Just a few case studies can be found in literature within the industrial environment and they show conflicting results (positive, negative and neutral). In this report, we present a successful application of UTDD in combination with ATDD in a commercial project. By successful we mean that the project goals are reached without an extra economic cost. All the UTDD and ATDD implementations are based on the same basic concepts, but they may differ in specific adaptations to each project or team. In the implementation presented here, the business requirements are specified by means of executable acceptance tests, which then are the input of a development process where the functional code is written in response to specific unit tests. Our goal is to share our successful experience in a specific project from an empirical point of view. We highlight the advantages and disadvantages of adopting UTDD and ATDD and identify some conditions that facilitate success. The main conclusions we draw from this project are that ATDD contributes to clearly capture and validate the business requirements, but it requires an extensive cooperation from the customer; and that UTDD has not a significant impact neither on productivity nor on software quality. These results cannot be generalized, but they point out that under some circumstances a test-driven development strategy can be a possible option to take into account by software professionals.

Design and industrial evaluation of a tool supporting semi-automated website testing

Abstract: SUMMARY Software testing is the most time-intensive and resource-intensive aspect of software development. Can support for testing be improved? This case study describes the motivations and design decisions behind the development of the testing tool, CoTester and its deployment to multiple development teams. CoTester outperforms available testing tools by representing tests using an easy-to-understand scripting language and thus making the tests easily editable. The design decisions of the testing tool were derived after conducting a series of interviews with testers and collecting their experiences with manual as well as automated testing. CoTester was developed to support these users, working in an environment of mixed manual and automatic tests, with a progression from manual to automatic testing when circumstances warrant. A series of deployments to four development teams showed that CoTester worked very well for non-professional testers (i.e. those who do testing only part-time), and it was also found to be useful by some professional testers. Copyright © 2012 John Wiley & Sons, Ltd.

Keyword driven automated testing framework for web application

Abstract: Presently in software testing companies, 90% of software testing is carried out manually which takes lots of time, and also need large number of employees which not only increase the cost of the Software but also takes more time for completion. Therefore, software testing companies are moving towards automated testing and there is a need to develop a testing framework which is application independent and scalable with the increased requirements of each application. An automated testing framework has been proposed here for testing web applications. The proposed framework is based on keyword driven framework in which a excel file containing the keywords is imported to the framework by using selenium WebDriver API.

Ambiguity Acceptance Testing : A Comparison of the Ratio Test and Difference Test

Abstract: Integer ambiguity resolution is an indispensable procedure for all high precision GNSS applications. The correctness of the estimated integer ambiguities is the key to achieving highly reliable positioning, but the solution cannot be validated with classical hypothesis testing methods. The integer aperture estimation theory unifies all existing ambiguity validation tests and provides a new prospective to review existing methods, which enables us to have a better understanding on the ambiguity validation problem. This contribution analyses two simple but efficient ambiguity validation test methods, ratio test and difference test, from three aspects: acceptance region, probability basis and numerical results. The major contribution of this paper can be summarized as: (1) The ratio test acceptance region is overlap of ellipsoids while the difference test acceptance region is overlap of half-spaces. (2) The probability basis of these two popular tests is firstly analyzed. The difference test is an approximation to optimal integer aperture, while the ratio test follows an exponential relationship in probability. (3) The limitations of the two tests are firstly identified. The two tests may under-evaluate the failure risk if the model is not strong enough or the float ambiguities fall in particular region. (4) Extensive numerical results are used to compare the performance of these two tests. The simulation results show the ratio test outperforms the difference test in some models while difference test performs better in other models. Particularly in the medium baseline kinematic model, the difference tests outperforms the ratio test, the superiority is independent on frequency number, observation noise, satellite geometry, while it depends on success rate and failure rate tolerance. Smaller failure rate leads to larger performance discrepancy.

Experience the relevance of testing in engineering design education

Abstract: Engineering design education imparts students how to Conceive, Design, Implement and Operate complex engineering systems. The „Innovation Project“ at ETH Zurich is a projectbased engineering design course, where 456 undergraduate students of mechanical engineering experience the CDIO activities in the freshman year in teams of 4 to 6 students. Starting from an idea, a concept and a design, the project work includes the production of several prototypes with direct manufacturing technologies and the implementation of the final system in a defined operation environment. A main finding of the project in 2013 is that systems being tested at different stages of the product development process showed higher reliability and better performance in the final contest. As a consequence, the learning effects of testing have considerable impact on the projects’ success. This paper shows how the relevance of testing can be experienced in project-based engineering design education focusing on the case study of the “Innovation Project” where testing is approached in all four CDIO activities: (1) Testing in Conceiving by the development of low-fidelity prototypes facilitates the exploration of ideas and the verification of the concept. (2) Testing in Design enables the students to identify errors such as part collisions primarily in the context of Computer Aided Design. (3) Testing the Implementation refers to testing the assembly of the parts. Finally, (4) Testing for Operation is conducted by integrating the system in the intended overall system. Based on the results, it is discussed how the relevance of testing can be experienced and imparted in project based engineering design classes.

A Review of Scripting Techniques Used in Automated Software Testing

Abstract: Software testing is the process of evaluating the developed system to assess the quality of the final product. Unfortunately, software-testing process is expensive and consumes a lot of time through software development life cycle. As software systems grow, manual software testing becomes more and more difficult. Therefore, there was always a need to decrease the testing time. Recently, automation is as a major factor in reducing the testing effort by many researchers. Therefore, automating software-testing process is vital to its success. This study aims to compare the main features of different scripting techniques used in process of automating the execution phase in software testing process. In addition, an overview of different scripting techniques will be presented to show the state of art of this study.

Industrial Application of Visual GUI Testing: Lessons Learned

Abstract: A large body of academic knowledge has been devoted to automated software testing in order to support the software market’s demands for continuous software delivery. However, most of these automated techniques approach testing from lower levels of system abstraction, e.g., component level, which limit their applicability for high-level regression testing of, for instance, system and acceptance tests, thus forcing companies to perform these test activities manually, which is considered time consuming, tedious, and error prone.

Industrial automation social media paradigms

Abstract: An industrial social media platform facilitates sharing of design, maintenance, and operational information relating to a planned or installed industrial control system. The industrial social media platform shares information between personnel involved in the design and testing of the control system, including plant engineers, original equipment manufacturers, system integrators, maintenance personnel, and equipment operators. The industrial social media platform creates an industrial social media page for a new control system prior to or during the design phase. Subscribers to the industrial social media page can share relevant information during design and testing of the industrial system, including design specifications, preferred engineering standards, schedules, acceptance test results, etc. After the industrial control system is deployed, the industrial social media page can be re-configured to share data relating to operation and maintenance of the system.

Persona Based Accessibility Testing

Abstract: Web authors have a hard time understanding and applying accessibility guidelines. The guidelines are considered too technical, without providing sufficient support for problem solving. This results in bad usability of Web applications for people who rely on accessibility. In the field of designing Web applications and interfaces, the concept of personas, as a representation of the target audience, is well established. Personas are typically used to describe the user on a personal level, with their needs, preferences and habits. In this poster, we illustrate a new workflow approach for accessibility evaluations. We propose persona-based representations of accessibility guidelines for acceptance tests of Web applications, for web authors to gain understanding on the needs of people with disabilities and thus improve the accessibility of Web applications.

Towards the use of an automatic assessment system in the teaching of software testing

Abstract: Both software testing and programming foundations are difficult subjects to be taught. In case of programming, experiences have suggested that the mastering of testing can contribute to enhance the students' skills of comprehension and analysis. So, automatic test-based assessment systems have been developed and adopted in programming courses. Aiming at investigating the adoption of assessment systems also in the teaching of software testing, in this paper we discuss the use of ProgTest — a tool for the submission and automatic evaluation of programming assignments based on testing activities. As most of the automatic assessment systems, ProgTest offers benefits such as immediate feedback, objectivity and consistency of the evaluation, and a substantial time saving in the evaluation of the programming assignments. Additionally, since it performs the assessments based on testing activities, ProgTest also offers benefits for the teaching of software testing. Results from two experiments involving the use of ProgTest in testing courses are presented, providing evidences on the feasibility of its adoption to support the teaching of testing concepts and criteria.