Showing papers on "Acceptance testing published in 2005"

Software penetration testing

Abstract: Quality assurance and testing organizations are tasked with the broad objective of assuring that a software application fulfills its functional business requirements. Such testing most often involves running a series of dynamic functional tests to ensure proper implementation of the application's features. However, because security is not a feature or even a set of features, security testing doesn't directly fit into this paradigm

Methodological Issues in Model-Based Testing

Abstract: Testing denotes a set of activities that aim at showing that actual and intended behaviors of a system differ, or at increasing confidence that they do not differ. Often enough, the intended behavior is defined by means of rather informal and incomplete requirement specifications. Test engineers use these specification documents to gain an approximate understanding of the intended behavior. That is to say, they build a mental model of the system. This mental model is then used to derive test cases for the implementation, or system under test (SUT): input and expected output. Obviously, this approach is implicit, unstructured, not motivated in its details and not reproducible.

Method and system for automated distributed software testing

Abstract: Embodiments of the present invention provide methods and systems for automated distributed testing of software. A testing service may utilize a distributed architecture that provides a standardized framework for writing tests, scheduling the tests, and gathering and reporting results of the tests. Multiple distributed labs are integrated into the testing service and their environments can be centrally managed by the testing service. The testing service permits the scheduling and performance of tests across multiple machines within a test lab, or tests that span across multiple test labs. Any of the machines in the test labs may be selected based on variety of criteria. The testing service may then automatically locate the appropriate machines that match or satisfy the criteria and schedule the tests when the machines are available. The testing service may also automatically create and set up a desired test environment according to the desired specifications for the test.

How to Do Hardware-in-the-Loop Simulation Right

Abstract: Not only is the number of electronic control units (ECUs) in modern vehicles constantly increasing, the software of the ECUs is also becoming more complex. Both make testing a central task within the development of automotive electronics. Testing ECUs in real vehicles is time-consuming and costly, and comes very late in the automotive development process. It is therefore increasingly being replaced by laboratory tests using hardware-in-the-loop (HIL) simulation. While new software functions are still being developed or optimized, other functions are already undergoing certain tests, mostly on module level but also on system and integration level. To achieve the highest quality, testing must be done as early as possible within the development process. This paper describes the various test phases during the development of automotive electronics (from single function testing to network testing of all the ECUs of a vehicle). The requirements for the test system and corresponding concepts are described. The paper also focuses on test methods and technology, and on the options for anchoring HIL simulation into the development process. HARDWARE-IN-THE-LOOP SIMULATION: AN ESTABLISHED PART OF THE CONTROL DEVELOPMENT PROCESS Time to market is speeding up, especially in automotive electronics. 90% of automotive innovations are currently connected with new electronics. Test drives can scarcely cope with the volume of systematic testing needed, especially just before start of production. The growing number of recall campaigns is a clear indication of this. It is little wonder that testing and error finding have become key tasks in the development process. [5] ECU testing typically is done using hardware-in-the-loop simulation. The ECU (prototype) is connected to a realtime simulation system simulating the plant (engine, vehicle dynamics, transmission, etc.) or even the whole vehicle. One means of reducing development times is to schedule early availability of the test system, which can be achieved by integrating HIL into the development process and involving the HIL system supplier as soon as ECU specification is available. This allows the simulator to be up and running shortly after receipt of A-, B-, and C-sample ECUs. Automated tests increase test coverage and shorten testing times by running complete test suites and overnight tests. HIL systems testing 24 hours, 7 days per week are not fiction but reality. Another measure taken by the OEMs is to transfer testing responsibility to the suppliers. Nowadays suppliers are more and more forced to perform early HIL tests. This not only includes function tests during function design but also complete integration and acceptance tests. The need for suppliers and OEMs to exchange tests, test results, models, etc., is important in this context. DIFFERENT USERS DIFFERENT NEEDS As HIL has become a standard method for testing ECUs and control strategies during the whole development cycle (i.e., not only after availability of the final ECUs), different needs of different users have to be addressed by the various test systems. Figure 1 shows the various HIL applications and the resulting test contents of the different phases. Figure 1.: Usage of HIL during the development process. FUNCTION DEVELOPER AT ECU SUPPLIER (OR AT OEM) Typically, only prototype ECUs are available during function development. Microscopic tests on a function are essential at this stage. The control strategy itself needs to be validated. Flexible, interactive operation needs to be possible. The simulator hardware also needs to be flexible for easy adaptation to changes in the ECUs or its peripherals. Low automation is typically required. During this development phase, test scripts are often set up in parallel to ECU/function development, or even after ECU/function development has finished. Even though the diagnostics procedure must also be tested, it might be necessary to deactivate some diagnostics of the ECU during function testing. The diagnostics depend on signal values that have to be calibrated. Often the calibration of the ECU functions is done prior to calibrating the diagnostics, which necessitates deactivation.. The typical objective of this phase is function acceptance testing. Ideally, this is automated by running the test scripts for all modules. Reusing control functions for different OEMs requires flexible HIL systems that can be adapted to the different ECU variants. Administration of the HIL software components, such as partial models and test scripts, is also needed. To avoid redundancy, tests successfully performed during function development should not have to be repeated during integration testing. While at this stage, functions are verified by HIL tests, it is important to test the proper interaction of all functions during integration testing. Close cooperation between supplier and OEM is therefore desirable, to exchange test protocols on the one hand and models (which are typically available at the OEM) on the other. In the case of function development at the OEMs (which typically still requires function integration into the final ECU provided by a supplier), the test process is quite similar to the process described above. Exchanging models and tests is far easier, however, as the exchanged data remains under the same roof. ECU PROJECT MANAGER AT THE OEM (OR SUPPLIER) – RELEASE AND ACCEPTANCE TEST Once all the functions have been integrated together with the lower software levels (operating system, I/O drivers), macroscopic testing of the complete ECU and/or its functions needs to be performed. This includes tests on overlapping administration layers (handling of diagnostic memory). Either the manufacturer or supplier performs an ECU release test. Automated tests are indispensable at this stage. The HIL should only be used interactively to find the cause in the event of an unexpected error. Manufacturers definitely need to repeat tests on ECUs that are provided by different suppliers (second source). Flexible systems that can be adapted to various ECU types are required at this stage. However, the administration of the simulator’s software components (partial models, test scripts, etc.) is even more important. Experiment software layouts represent the functionality of the test system. The objective is to release the complete ECU as errorfree including diagnostics. VEHICLE ELECTRONICS SYSTEM MANAGER AT THE OEM As already mentioned, tests that were already finished on component level should not be repeated when networked systems are tested, for efficiency reasons. In an examination of release tests for the complete vehicle electronics, the focus lies explicitly on testing distributed functions and testing bus communication. Network management is also one function under test in this context. Another important issue comes into play at this stage, if it has not already done so: variant handling. Countryspecific variants for a worldwide market presence, as well as different equipment variants and frequent revisions in model cycles, make it necessary to handle different configurations. As a result, combinatorial tests for the various ECU/vehicle variants are required. This again requires a flexible system based on hardware and software that support different variants in plant models, I/O channels, and bus communication. Automated tests are indispensable here. Tests designed for the system can easily be replayed for all ECU/vehicle variants. The higher the degree of automation, the higher the test coverage. Only a few of the tests established on function level should be reused here. Another important aspect of automated tests is that tests, which verified performance during the development phase, can also be used to investigate warranty issues after series production has started. The complete system must be error-free including diagnostics. WHAT NEEDS TO BE CONSIDERED WHEN CONFIGURING/SELECTING A SIMULATOR? Instead of being connected to an actual vehicle, the electronic control unit(s) to be tested are connected to a hardware-in-the-loop simulation system. Software and hardware models simulate the behavior of the vehicle and related sensors and actuators. The models were typically developed with a suitable modeling tool, such as MATLAB®/Simulink®. C code is generated automatically and downloaded to real-time processors for execution. I/O boards, together with signal conditioning for level adaptation to the automotive voltages required by the ECU, provide the interface to the ECU pins. Figure 2 shows a typical hardware-in-the-loop system architecture [1]. The most important components of an HIL system are described below. Figure 2.: Typical hardware-in-the-loop system architecture.

Multi-agent-based integrated framework for intra-class testing of object-oriented software

Abstract: The primary features of the object-oriented paradigm lead to develop a complex and compositional testing framework for object-oriented software. Agent-oriented approach has become a trend in software engineering. Agent technologies facilitate the software testing by virtue of their high-level independency with parallel activation and automation. This paper proposed an integrated framework that has been built on two existing testing techniques namely Mutation Testing and Capability Testing. In both the cases, testing is carried out at Autonomous Unit Level (AUL) and Inter-Procedural Level (IPL). Mutation-Based Testing-Agent and Capability Assessment Testing-Agent have been developed for performing AUL testing and Method Interaction Testing-Agent has been developed for performing IPL testing. This agent-based framework is an attempt for developing an automated software testing environment and among the several phases of Software Development Life Cycle (SDLC), this framework is recommended for unit testing in code complete phase and alpha phase. This methodology gives the basic approach to agent-based frameworks for testing and to optimization of agent-based testing schedules, subject to timing constraints. This adds ''interesting new opportunities in the object-oriented software testing phase'' to the existing literature that is concerned with software testing frameworks.

A standard approach to measurement uncertainties for scientists and engineers in medicine.

Abstract: The critical nature of health care demands high performance levels from medical equipment. To ensure these performance levels are maintained, medical physicists and biomedical engineers conduct a range of measurements on equipment during acceptance testing and on-going quality assurance programs. Wherever there are measurements, there are measurement uncertainties with potential conflicts between the measurements made by installers, owners and occasionally regulators. Prior to 1993, various methods were used to calculate and report measurement uncertainties. In 1993, the International Organization for Standardization published the Guide to the Expression of Uncertainty in Measurement (GUM). The document was jointly published with six international organizations principally involved in measurements and standards. The GUM is regarded as an international benchmark on how measurement uncertainty should be calculated and reported. Despite the critical nature of these measurements, there has not been widespread use of the GUM by medical physicists and biomedical engineers. This may be due to the complexity of the GUM. Some organisations have published guidance on the GUM tailored to specific measurement disciplines. This paper presents the philosophy behind the GUM, and demonstrates, with a medical physics measurement example, how the GUM recommends uncertainties be calculated and reported.

UCM-Driven testing of web applications

Abstract: Despite their apparent simplicity, Web applications are surprisingly difficult to develop, if our aim is to build applications that behave correctly under regular conditions as well as adverse circumstances like out-of-order requests and race conditions. In this paper, we describe our experiences deriving customer-oriented acceptance tests for Web applications by modeling the essential capabilities of such applications with Use Case Maps (UCMs). Abstract test purposes are generated from a UCM model using scenario definitions and scenario extraction tools. These test purposes are then converted interactively to test cases in the FitNesse acceptance testing framework, which is popular in the Extreme Programming (XP) community. The test cases are used to validate a Web application where several typical but non-trivial bugs were planted. Challenges in the automation of the process are also discussed.

Finding and ranking research directions for software testing

Abstract: The percentage of software testing in software development is high; 50 % is often mentioned in the literature. Moreover, the amount of testing is increasing because of the demand for better quality. A survey was carried out to find out where software testing research should be directed. The experts of the participating industry and the research institutes evaluated the issues of testing research. The research method was a derivative of the Delphi method. The most prominent research issues in ranking included process improvement, testing automation with testing tools, and standardization of testing. The process and results of the survey are presented and the need for further research is discussed. This study also shows an example how project steering groups can be used in resolving research priorities.

Research in testing COTS components - built-in testing approaches

Abstract: Summary form only given. The use of components in the development of large software systems can have various benefits. Their testing, however, is still an open issue in software engineering. The user of a component is often faced with the problem that information necessary for testing purposes is not available. This lack of information distinguishes the testing of components from other software entities known from non-component-based development. This article gives an overview of the approaches to testing components. Besides the overview, this article also outlines some of the limitations of these approaches. Testing components is still an open problem.

Proof vs testing in the context of safety standards

Abstract: During software certification various forms of testing (e.g., unit, integration, regression) are undertaken. These testing processes are very important, but are also generally accepted as expensive, leading to a desire to replace testing with more cost-effective processes, where practicable. This paper is concerned with how such technology substitution can be justified, and presents a template for an argument that can be used to justify substitutions. It also instantiates the argument for a particular proof technology - the CLawZ toolset - and demonstrates how to argue for its safe substitution for testing in this context.

The test bus imperative: architectures that support automated acceptance testing

Abstract: Agile methods more specifically, test-driven development practices have begun to raise the software industry's awareness of automated acceptance testing. Many tools can be purchased to help testers write automatic scripts for testing the system through its user interface. Unfortunately, testing through the UI is slow, opaque, and dangerous. The software industry's increased awareness of automated testing benefits and potentials is good new for an industry that for decades has been beset by quality and productivity issues.

In Situ Load Testing of Parking Garage Reinforced Concrete Slabs: Comparison between 24 h and Cyclic Load Testing

Abstract: This paper reports on the results obtained on the applicability of the diagnostic cyclic load test method in comparison with the existing 24 h test procedure adopted in ACI-318. A parking garage, owned by St. Louis County, St. Louis, Missouri, scheduled for demolition during the summer of 2002 was used as a research test bed before demolition. This structure, a two-story steel and reinforced concrete (RC) frame with one-way RC slabs built in 1970's, was ideal, in terms of size and construction system, for performing comparative field experimentation on load testing. Investigation and validity of acceptance criteria of existing and proposed testing methods were performed. Two identical RC slabs were tested, according to both the standard procedure (ACI-318-02) and the proposed diagnostic load testing. In both instances, the applied total test load was such that the slab did not pass the load test. This allowed characterizing the critical test parameters that govern acceptability and draw conclusions on their values. After load testing, both slabs were loaded until partial collapse was reached. This allowed making observations on the margin of safety with respect to collapse, a determination that is not generally possible in a proof test. The analysis of the experimental data provides professionals with evidence on the validity of in situ assessment for the adequacy of structural members.

An environment for training computer science students on software testing

Abstract: Software testing is essential to ensure software quality. On most software projects testing activities consume at least 30 percent of the project effort. On safety critical applications, software testing can consume between 50 to 80 percent of project effort. There is a vast amount of information available on software testing techniques and tools and some universities offer software testing courses at the advanced undergraduate and graduate level. Software testing must also be stressed in beginning courses and even in the high schools when students are first learning to program. It is also important in these early years to instill a respect for software testing and some interest in possible testing careers. This paper describes a project to develop a learning and training environment that enables students to develop the knowledge and skills to perform requirement based testing. The target audience for the environment are beginning programming students at either the high school or university level. The software training environment consists of Web-based instructional materials as well as a testing simulator which enables students to actually test software programs. This paper describes the educational objectives of the test training environment, its implementation and results from utilizing the environment with beginning programming students

Software reliability improvement through operational profile testing

Abstract: In this paper a software which had undergone normal scheduled testing cycle of the company before is taken up for testing through operational profile. Operational profile of the software is developed and test cases based on the operational profile are allocated. Issue of allocation of test cases to infrequent operations is also resolved. Testing driven by operational profile is found to be effective and practical as some critical failures are observed during testing. Failure data is analyzed, reliability growth models are applied and estimates of software reliability and failure intensity are made. Software reliability demonstration chart based on consumer risk, producer risk and discrimination ratio is also prepared. Failure data of certification testing is plotted on the chart to arrive at the decision of accepting/rejecting the software with a small sample size of failure data.

The practice of specifying requirements using executable acceptance tests in computer science courses

Abstract: This report describes the practice of using executable acceptance testing for specifying programming assignments in software engineering courses. We summarize experiences from two courses introduced in two academic institutions over four semesters -- both from students' and instructors' perspectives. Examples of projects and the discussion of the assignment flows are given. The paper highlights testing as an all-encompassing activity in software development projects. It also contains recommendations for academics thinking of incorporating executable acceptance testing into their courses.

Examining usage patterns of the FIT acceptance testing framework

Abstract: Executable acceptance testing allows both to specify customers' expectations in the form of the tests and to compare those to actual results that the software produces. The results of an observational study identifying patterns in the use of the FIT acceptance testing framework are presented and the data on acceptance-test driven design is discussed.

Method and system for drug screening

Abstract: The invention provides a system and method for screening drugs from candidate compounds selected from a library. The system includes multiple hardware components and a computer software system for scheduling and coordinating the operations of the hardware components. A user of the system requests a series of assays to be performed on the candidate compounds. Each assay is associated with a test acceptance criteria. Interdependencies of these assays may be specified. The software system schedules the hardware components to run each assay with minimum hardware idle time possible based on the assays requested and the interdependencies of these assays specified. The software system coordinates and directs the flow of samples and data through the system. A decision whether a sample can proceed to the next assay as scheduled may be made automatically based on test acceptance criteria, or manually modified by the user. All assays can be performed in an automated fashion once the samples are provided to the system.

System and method for selecting applicable tests in an automation testing system

Abstract: Automation test selection is made for an automation testing process according to the code changes that have occurred between builds of a software product. A mapping between the available tests and source tree locations of the files is maintained. The mapping allows the appropriate tests to be selected for the code changes that have been made, resulting in a more focused automation testing process.

Towards self-testing of component-based software

Abstract: It is widely accepted that conventional test methods are not necessarily adequate for testing of component-based software (CBS). As a consequence, also conventional test tools cause similar problems for the test automation of CBS based on their graphical user interfaces (GUI), because for any level of user-focused testing domain knowledge and knowledge about the implementation of the CBS are essential to run the tests. The component manufacturer, on the other side, is usually not willing to deliver the code to protect his, or her, commercial interest. For solving this conflict, this paper introduces a framework for the automation of user-oriented component testing that significantly reduces the test costs. The concept is based on black-box testing techniques and utilizes the common features of commercial capture/replay test tools.

The Software Testing Technique

Abstract: Before being submitted to the client, the software product should be taken a very comprehensive testing to ensure it can run reliably and stably. In the quality viewpoint, software testing is the most important means to guarantee the software quality.No matter to object-oriented programming or to procedure-oriented programming, software testing can only be fulfilled by a set of testing techniques and methods on earth. This paper expatiates 3 techniques: error-avoidance, error-check and fault-tolerance, and also introduces 2 methods: passable testing and failure testing.

A process-complete automatic acceptance testing framework

Abstract: We present a new automated software acceptance tests framework. The framework is novel in supporting the entire lifecycle and all QA activities, including test maintenance over multiple versions, interaction with programmers and business analysts, traceability to specifications, multi-user test cases and more. This enables a significant increase in QA productivity and product quality. We compare our framework to other available tools, products and frameworks, and present several patterns and anti-patterns for implementing a successful automated acceptance testing solution.

A framework for testing distributed software components

Abstract: Component-based software engineering is an influential trend in software engineering. Adopting component-based techniques, a system can be constructed by synthesis of various distributed components. This paper presents a framework of remote testing of distributed software components. Based on the CORBA architecture and Java technology, this paper provides an environment to allow a client-side software component to define tests for a black-box component published on the server-side. This technique simplifies test execution, test results check and report, and supports test reuse and test automation. The paper reveals a practical approach to test software components by enhancing software component testability and test re-usability. The incremental testing framework introduced in this paper is helpful in saving time, energy, and cost required for testing distributed components and for enhancing software quality. A testing supporting tool is implemented to facilitate distributed component testing based on CORBA

An evaluation of aspect oriented testing techniques

Abstract: Aspect oriented programming (R.T. Alexander, et al) promises to enhance software quality by increasing the cohesion of classes and localizing both core and crosscutting concerns. The quality of software, however, can only be validated by testing the software. Testing aspect oriented programs remains just as important as testing any other software. This paper presents an analysis of the testing strategies for AOPs. Three testing strategies have been examined and their effectiveness is measured in terms of their ability to find different kind of faults as described in a fault model by R.T. Alexander, et al. Based on this analysis, conclusions have been drawn about the current state of the research in the testing of aspect oriented programs and future directions have been explored.

Acceptance testing of actuators using predetermined thresholds

Abstract: An improved technique of acceptance testing an actuator system involves measuring system parameters of the actuator system during operation of the actuator system, and storing the measured system parameters in computerized memory as a set of measured system parameters. The technique further involves obtaining a set of predetermined thresholds. Each predetermined threshold corresponds to a particular measured system parameter. The technique further involves electronically indicating whether the actuator system is in acceptable condition based on an electronic comparison of the set of measured system parameters stored in the computerized memory and the set of predetermined thresholds, e.g., providing a warning of a need for service or of imminent failure when one or more of the system parameters exceeds a corresponding threshold.

Student experiences with executable acceptance testing

Abstract: This report describes experiences of introducing executable acceptance testing in senior software engineering courses. Students in an agile environment completed a five iteration project with significant portion of requirements specified as executable acceptance tests. Furthermore, students were required to write their test suites for an additional component. Ability to learn and utilize the FIT acceptance testing framework is evaluated to find out if FIT tests can be used to replace requirement documents. The results of a survey of students' perceptions and experiences were encouraging.

IEC 60034-18-41: a new draft technical specification for qualification and acceptance tests of inverter duty motor insulation

Abstract: A new International Electro-technical Commission (IEC) Technical Specification (TS) is being developed to define the requirements for qualification of insulation systems to be used in machines supplied by voltage converters. Currently being prepared, it is much more inclusive than NEMA MG-I part 31 (Definite-Purpose Inverter-Fed Polyphase Motors). Once approved, it will significantly influence the development of applicable North American specifications. The TS defines qualification and acceptance tests for inverter duty motor insulation systems It applies to random wound and form wound machines, and contains a very good informative section on drive-related insulation issues. The TS applies to insulation systems for bars and coils in both rotors and stators, but this paper addresses only stator windings. Many manufacturers today are working to develop insulation systems that will meet the stringent requirements of the Technical Specification. This paper presents the highlights of the draft document and how it relates to the current methods of specifying machine insulation systems with non-sinusoidal power supplies. Changes to the way inverter-duty insulation systems are qualified as a result of this specification is discussed and evaluated.

Design, Implementation, and Validation of KOMPSAT-2 Software Simulator

Abstract: In this paper, we present design features, implementation, and validation of a satellite simulator subsystem for the Korea Multi-Purpose Satellite-2 (KOMPSAT-2). The satellite simulator subsystem is implemented on a personal computer to minimize costs and trouble on embedding onboard flight software into the simulator. An object-oriented design methodology is employed to maximize software reusability. Also, instead of a high-cost commercial database, XML is used for the manipulation of spacecraft characteristics data, telecommand, telemetry, and simulation data. The KOMPSAT-2 satellite simulator subsystem is validated by various simulations for autonomous onboard launch and early orbit phase operations, anomaly operation, and science fine mode operation. It is also officially verified by successfully passing various tests such as the satellite simulator subsystem test, mission control element system integration test, interface test, site installation test, and acceptance test.

Prediction-Based software availability enhancement

Abstract: We propose a new paradigm for software availability enhancement. We offer a two-step strategy: Failure prediction followed by maintenance actions with the objective of avoiding impending failures or minimizing the effort of their repair. For the first step we present two failure prediction methods: universal basis functions (UBF) and similar events prediction (SEP), which are based on probabilistic analysis. The potential of the presented methods is evaluated by a case-study where failures of a commercial telecommunication platform have been predicted. The second step includes existing maintenance methods fitting the proposed approach and a new recovery strategy called “adaptive recovery blocks”. Since system availability enhancement is the overall goal, equations to calculate availability of such a system are given as well.