Showing papers on "White-box testing published in 2018"

DLFuzz: differential fuzzing testing of deep learning systems

Abstract: Deep learning (DL) systems are increasingly applied to safety-critical domains such as autonomous driving cars. It is of significant importance to ensure the reliability and robustness of DL systems. Existing testing methodologies always fail to include rare inputs in the testing dataset and exhibit low neuron coverage. In this paper, we propose DLFuzz, the first differential fuzzing testing framework to guide DL systems exposing incorrect behaviors. DLFuzz keeps minutely mutating the input to maximize the neuron coverage and the prediction difference between the original input and the mutated input, without manual labeling effort or cross-referencing oracles from other DL systems with the same functionality. We present empirical evaluations on two well-known datasets to demonstrate its efficiency. Compared with DeepXplore, the state-of-the-art DL whitebox testing framework, DLFuzz does not require extra efforts to find similar functional DL systems for cross-referencing check, but could generate 338.59% more adversarial inputs with 89.82% smaller perturbations, averagely obtain 2.86% higher neuron coverage, and save 20.11% time consumption.

Pengujian Aplikasi dengan Metode Blackbox Testing Boundary Value Analysis (Studi Kasus: Kantor Digital Politeknik Negeri Lampung)

Abstract: Software testing phase is one of a critical element in determining the quality of a software. These tests include design, specification, and coding. This study aims to test the digital office software at Lampung State Polytechnic. The testing process is done to determine the level of error that occurs in the software. The test used a black box testing Boundary Value Analysis. Boundary Value Analysis is a type of test case by determine the normal value, minimum value and maximum value of the tested data. The applications resulted from this research are capable to handling data, both normal and abnormal data with a 91, 67% success rate.

Uji Fungsionalitas (Blackbox Testing) Sistem Informasi Lembaga Sertifikasi Profesi (SILSP) Batik dengan AppPerfect Web Test dan Uji Pengguna

Abstract: Information systems in an organization have an important role in providing information to management in order to control various complex processes within an organization. The implementation of information systems in the Batik Professional Certification Institute, can help the management of Batik LSP in controlling processes in organizations, ranging from participants, assessors, Competency Test Places, to scheduling competency tests. Information systems designed to build must be through a test before use. Information system testing is carried out to determine whether or not the application is suitable for use by users. This study aims to conduct a series of tests on information systems that have been designed to build. The method used is experimental, using the AppPerfect Web Test application and the Functional Test questionnaire (Black Box Testing) instrument provided to users. The purpose of this study, to determine the extent to which these two test methods can be reliable in testing the information system of professional certification institutions that are built. Based on the analysis carried out, the AppPerfect Web Test application is in line with the results of the function test by the user. In other words, functional testing can be done by users or with the AppPerfect Web Test application, or both.

Fuzz Testing for Automotive Cyber-Security

Abstract: There is increasing computational complexity within the connected car, and with the advent of autonomous vehicles, how do manufacturers test for cyber-security assurance? The fuzz test is a successful black box testing method that hackers have used to find security weaknesses in various domains. Therefore, should the fuzz test, mentioned (without any details) in SAE J3061, be applied more widely into the vehicle systems development process to help reduce vulnerabilities? To investigate this question a custom fuzzer was developed to allow for experimentation against a target vehicle's CAN bus (used as the data interconnect for the vehicle's ECUs). The results demonstrate that the fuzz test has a part to play as one of the many security tests that a vehicle's systems need to undergo before being made ready for series production. However, previous problems raised when cyber testing a vehicle were confirmed. Thus, in adding the fuzz test to the automotive engineering tool box some issues are raised that need addressing in future research.

TSTL: the template scripting testing language

Abstract: A test harness, in automated test generation, defines the set of valid tests for a system, as well as their correctness properties. The difficulty of writing test harnesses is a major obstacle to the adoption of automated test generation and model checking. Languages for writing test harnesses are usually tied to a particular tool and unfamiliar to programmers, and often limit expressiveness. Writing test harnesses directly in the language of the software under test (SUT) is a tedious, repetitive, and error-prone task, offers little or no support for test case manipulation and debugging, and produces hard-to-read, hard-to-maintain code. Using existing harness languages or writing directly in the language of the SUT also tends to limit users to one algorithm for test generation, with little ability to explore alternative methods. In this paper, we present TSTL, the template scripting testing language, a domain-specific language (DSL) for writing test harnesses. TSTL compiles harness definitions into an interface for testing, making generic test generation and manipulation tools for all SUTs possible. TSTL includes tools for generating, manipulating, and analyzing test cases, including simple model checkers. This paper motivates TSTL via a large-scale testing effort, directed by an end-user, to find faults in the most widely used geographic information systems tool. This paper emphasizes a new approach to automated testing, where, rather than focus on developing a monolithic tool to extend, the aim is to convert a test harness into a language extension. This approach makes testing not a separate activity to be performed using a tool, but as natural to users of the language of the system under test as is the use of domain-specific libraries such as ArcPy, NumPy, or QIIME, in their domains. TSTL is a language and tool infrastructure, but is also a way to bring testing activities under the control of an existing programming language in a simple, natural way.

Inferring OpenVPN State Machines Using Protocol State Fuzzing

Abstract: The reliability of a security protocol is of the utmost importance but can easily be compromised by a vulnerability in the implementation. A crucial aspect of an implementation is the protocol's state machine. The state machine of an implementation can be inferred by black box testing using regular inference. These inferred state machines provide a good insight into implementations and can be used to detect any spurious behavior. We apply this technique to different implementations of OpenVPN: the standard OpenVPN and the OpenVPN-NL implementations. Although OpenVPN is a widely used TLS-based VPN solution, there is no official specification of the protocol, which makes it a particularly interesting target to analyze. We infer state machines of the server-side implementation and focus on particular phases of the protocol. Finally we analyze those state machines, show that they can reveal a lot of information about the implementation which is missing from the documentation, and discuss the possibility to include state machines in a formal specification.

Learning-based testing for autonomous systems using spatial and temporal requirements

Abstract: Cooperating cyber-physical systems-of-systems (CO-CPS) such as vehicle platoons, robot teams or drone swarms usually have strict safety requirements on both spatial and temporal behavior. Learning-based testing is a combination of machine learning and model checking that has been successfully used for black-box requirements testing of cyber-physical systems-of-systems. We present an overview of research in progress to apply learning-based testing to evaluate spatio-temporal requirements on autonomous systems-of-systems through modeling and simulation.

DLFuzz: Differential Fuzzing Testing of Deep Learning Systems

Abstract: Deep learning (DL) systems are increasingly applied to safety-critical domains such as autonomous driving cars. It is of significant importance to ensure the reliability and robustness of DL systems. Existing testing methodologies always fail to include rare inputs in the testing dataset and exhibit low neuron coverage. In this paper, we propose DLFuzz, the frst differential fuzzing testing framework to guide DL systems exposing incorrect behaviors. DLFuzz keeps minutely mutating the input to maximize the neuron coverage and the prediction difference between the original input and the mutated input, without manual labeling effort or cross-referencing oracles from other DL systems with the same functionality. We present empirical evaluations on two well-known datasets to demonstrate its efficiency. Compared with DeepXplore, the state-of-the-art DL whitebox testing framework, DLFuzz does not require extra efforts to find similar functional DL systems for cross-referencing check, but could generate 338.59% more adversarial inputs with 89.82% smaller perturbations, averagely obtain 2.86% higher neuron coverage, and save 20.11% time consumption.

An acceptance testing approach for Internet of Things systems

Abstract: Internet of things (IoT) systems are becoming ubiquitous and assuring their quality is fundamental. Unfortunately, a few proposals for testing these complex, and often safety-critical, systems are present in the literature. The authors propose an approach for acceptance testing of IoT systems adopting graphical user interfaces as a principal way of interaction. Acceptance testing is a type of black box testing based on test scenarios, i.e. sequences of steps/actions performed by the user or the system. In their approach, test scenarios are derived from a state machine that expresses the behaviour of the system under test, and test cases are derived from them by specifying the actual data and assertions and made executable by implementing the corresponding test scripts. As a case study, they selected a mobile health IoT system for diabetes management composed of local sensors/actuators, smartphones, and a remote cloud-based system. The effectiveness of the approach has been evaluated by measuring the capability of two test suites implemented using different localisation strategies (visual and structure-based) in detecting mutants of the original m-health system. Results show the effectiveness of the test suites implemented by following the proposed approach since 93% of the generated mutants have been detected.

Integrating software quality models into risk-based testing

Abstract: Risk-based testing is a frequently used testing approach which utilizes identified risks of a software system to provide decision support in all phases of the testing process. Risk assessment, which is a core activity of every risk-based testing process, is often done in an ad hoc manual way. Software quality assessments, based on quality models, already describe the product-related risks of a whole software product and provide objective and automation-supported assessments. But so far, quality models have not been applied for risk assessment and risk-based testing in a systematic way. This article tries to fill this gap and investigates how the information and data of a quality assessment based on the open quality model QuaMoCo can be integrated into risk-based testing. We first present two generic approaches showing how quality assessments based on quality models can be integrated into risk-based testing and then provide the concrete integration on the basis of the open quality model QuaMoCo. Based on five open source products, a case study is performed. Results of the case study show that a risk-based testing strategy outperforms a lines of code-based testing strategy with regard to the number of defects detected. Moreover, a significant positive relationship between the risk coefficient and the associated number of defects was found.

TESTOR: A Modular Tool for On-the-Fly Conformance Test Case Generation

Abstract: We present TESTOR, a tool for on-the-fly conformance test case generation, guided by test purposes. Concretely, given a formal specification of a system and a test purpose, TESTOR automatically generates test cases, which assess using black box testing techniques the conformance to the specification of a system under test. In this context, a test purpose describes the goal states to be reached by the test and enables one to indicate parts of the specification that should be ignored during the testing process. Compared to the existing tool TGV, TESTOR has a more modular architecture, based on generic graph transformation components, is capable of extracting a test case completely on the fly, and enables a more flexible expression of test purposes, taking advantage of the multiway rendezvous. TESTOR has been implemented on top of the CADP verification toolbox, evaluated on three published case-studies and more than 10000 examples taken from the non-regression test suites of CADP.

Conventional Software Testing Using White Box Method

Abstract: Software development process highly relates to analysis, design, coding, testing and implementation processes. Testing process becomes imperative process to maintain a quality product running well. Testing process can be conducted both for structural and object-oriented software. However, the method utilized for structural and object-oriented software is significantly different. Testing structural program can utilize White Box, Black Box, or Gray testing methods. This study White Box Testing has been employed to test a simple application. The testing process using White Box Testing employs some testing techniques based on path testing consisting of some processes, namely testing independent path, developing flow graph, calculating cyclomatic complexity, and developing graph matrices. Hence, the testing process employing White Box method with basis path testing technique can be executed.

Enterprise Architecture Planning in developing A planning Information System: a Case Study of Semarang State University

Abstract: This research has applied an integrated design and development of planning information system, which is been designed using Enterprise Architecture Planning. Frequent discrepancy between planning and realization of the budget that has been made, resulted in ineffective planning, is one of the reason for doing this research. The design using EAP aims to keep development aligned and in line with the strategic direction of the organization. In the practice, EAP is carried out in several stages of the planning initiation, identification and definition of business functions, proceeded with architectural design and EA implementation plan that has been built. In addition to the design of the Enterprise Architecture, this research carried out the implementation, and was tested by several methods of black box and white box. Black box testing method is used to test the fundamental aspects of the software, tested by two kinds of testing, first is using User Acceptance Testing and the second is using software functionality testing. White box testing method is used to test the effectiveness of the code in the software, tested using unit testing. Tests conducted using white box and black box on the integrated planning information system, is declared successful. Success in the software testing can not be ascertained if the software built has not shown any distinction from prior circumstance to the development of this integrated planning information system. For ensuring the success of this system implementation, the authors test consistency between the planning of data and the realization of prior-use of the information system, until after-use information system. This consistency test is done by reducing the time data of the planning and realization time. From the tabulated data, the planning information system that has been built reduces the difference between the planning time and the realization time, in which indicates that the planning information system can motivate the planner unit in realizing the budget that has been designed. It also proves that the value chain of the information planning system has brought implications for budget realization.

Is this the lifecycle we really want?: an automated black-box testing approach for Android activities

Abstract: Android is today the world's most popular mobile operating system and the demand for quality to Android mobile apps has grown together with their spread. Testing is a well-known approach for assuring the quality of software applications but Android apps have several peculiarities compared to traditional software applications that have to be taken into account by testers. Several studies have pointed out that mobile apps suffer from issues that can be attributed to Activity lifecycle mishandling, e.g. crashes, hangs, waste of system resources. Therefore the lifecycle of the Activities composing an app should be properly considered by testing approaches. In this paper we propose ALARic, a fully automated Black-Box Event-based testing technique that explores an application under test for detecting issues tied to the Android Activity lifecycle. ALARic has been implemented in a tool. We conducted an experiment involving 15 real Android apps that showed the effectiveness of ALARic in finding GUI failures and crashes tied to the Activity lifecycle. In the study, ALARic proved to be more effective in detecting crashes than Monkey, the state-of-the practice automated Android testing tool.

Testing closed source software: computer forensic tool case study

Abstract: Computer forensic techniques are important for the prevention, detection, and investigation of electronic crime. Computer forensic investigators need computer forensic tools to produce reliable results that meet legal requirements and are acceptable in the courts. Most of these tools are closed-source, making the software a black-box for testing purposes. This paper illustrates a different black box testing method for experimenting computer forensic tools based on functional scenarios.

Parameter optimization in control software using statistical fault localization techniques

Abstract: Embedded controllers for cyber-physical systems are often parameterized by look-up maps representing discretizations of continuous functions on metric spaces. For example, a non-linear control action may be represented as a table of pre-computed values, and the output action of the controller for a given input computed by using interpolation. For industrial-scale control systems, several man-hours of effort are spent in tuning the values within the look-up maps. %and sub-optimal performance is often associated with %inappropriate values in look-up maps. Suppose that during testing, the controller code is found to have sub-optimal performance. The parameter fault localization problem asks which parameter values in the code are potential causes of the sub-optimal behavior. We present a statistical parameter fault localization approach based on binary similarity coefficients and set spectra methods. Our approach extends previous work on (traditional) software fault localization to a quantitative setting where the parameters encode continuous functions over a metric space and the program is reactive. We have implemented our approach in a simulation workflow for control systems in Simulink. Given controller code with parameters (including look-up maps), our framework bootstraps the simulation workflow to return a ranked list of map entries which are deemed to have most impact on the performance. On a suite of industrial case studies with seeded errors, our tool was able to precisely identify the location of the errors.

Guided test case generation through AI enabled output space exploration

Abstract: Black-box software testing is a crucial part of quality assurance for industrial products. To verify the reliable behavior of software intensive systems, testing needs to ensure that the system produces the correct outputs from a variety of inputs. Even more critical, it needs to ensure that unexpected corner cases are tested. Existing approaches attempt to address this problem by the generation of input data to known outputs based on the domain knowledge of an expert. Such input space exploration, however, does not guarantee an adequate coverage of the output space as the test input data generation is done independently of the system output. The paper discusses a novel test case generation approach enabled by neural networks which promises higher probability of exposing system faults by systematically exploring the output space of the system under test. As such, the approach potentially improves the defect detection capability by identifying gaps in the test suite of uncovered system outputs. These gaps are closed by automatically determining inputs that lead to specific outputs by performing backward reasoning on an artificial neural network. The approach is demonstrated on an industrial train control system.

Edroid: A Mutation Tool for Android Apps

Abstract: Android applications have experienced an outstanding growth in recent years with a projected figure of 378 billion in the number of application downloads by 2020. As a result, verification and validation of Android apps and their graphical user interface through testing is crucial for quality assurance. One way to assess quality assurance is through the use of mutation testing. Mutation testing modifies the code of the System Under Test in small ways in order to produce faulty versions of the program called mutants and compare them to the original version. However, the cost of mutation testing can be expensive and time consuming. We propose a graphical user-friendly Android mutation tool called Edroid whose primary purpose is to mutate Android's main components such as activities, services, content providers and broadcast receivers using the source code of XML files. We introduce fourteen Android mutation operators, where ten of them are new, that are derived from operators used in different fields such as mutation testing for Graphical User Interface and XML schema. We evaluate Edroid on five real world Android applications using testing methodologies such as random testing and black box testing. Results from these experiments demonstrate that Edroid facilitates the generation of mutants. In addition and equally important, the set of proposed mutation operators are effective in revealing errors that improve the quality of a given test suite.

White-Box Testing Strategy for a Solar Tracking Device Using NodeMCU Lua ESP8266 Wi-Fi Network Development Board Module

Abstract: This paper presents a novel technique in testing the software code of a solar tracking device by implementing a White-box testing approach that makes use of a Wi-Fi module. First, we succeed in verifying if the wireless data transfer controlling the movements of the solar tracking device are in correspondence with the software code run on the main control board called Arduino UNO. Additionally to the local implementation, a cloud-based solution together with a mobile application to remotely control, test and communicate with the solar tracking device is proposed. Second, we used White-box testing techniques to test and give details about software errors in our solar tracker. We implemented unit testing techniques as well as custom code in order to find out all the loopholes and possible breakpoints in our solar tracker software by investigating Communication, Control Flow and Error handling errors. The experimental results show that our White-box testing strategy is efficient from the fault coverage and cost points of view.

Parallel testing and coverage analysis for context-free applications

Abstract: Software testing being one of the major phases of the software development life cycle is critical in delivering reliable software products. Traditional manual GUI testing has severe limitations like insufficient test coverage, labor intensiveness, complexity involved and cost ineffectiveness. “How much testing is enough or sufficient?” still remains a challenging question. Coverage analysis helps to guide the test engineers regarding test coverage and is used extensively to determine the effectiveness of selected testing methodology. The problems mentioned above related to manual GUI testing prompted the need for automation of GUI testing and coverage analysis. With the rapid emergence of GUI based context free applications, automated testing tools seems less effective. Recently, it has been learnt that concurrent and distributed machines based cluster can be used to reduce the required effort to test GUI in context of time. Proposed system use two different evolutionary algorithms concurrently to gain multiple objectives, based on the fitness functions of maximization of GUI test path coverage (measured as the number of events in an event sequence tested by a particular test case) and simultaneously minimizing the number of test cases. For coverage analysis, two of the very well-known multi objective algorithms; NSGA II and MOPSO from evolutionary domain have been employed. Experiments results of coverage analysis show that accuracy to the tune of 85% was achieved in MOPSO and NSGA II. This high level of accuracy is an indicator of usefulness of proposed fitness function. Both algorithms execute more than 90% of test paths.

Variable Strength Combinatorial Test Data Generation Using Enhanced Bird Swarm Algorithm

Abstract: combinatorial testing is an effective black box testing technique for the system with large numbers of parameters and their values. However, for significantly complex and key systems, combinatorial testing still owns high complexity. Testing completely the core part of these systems usually is a kind of solution and Variable strength combinatorial test data generation (VS-CTDG) emerges. In this paper, enhanced bird swarm algorithm (EBSA), a variant of Bird Swarm Algorithm (BSA), is employed into the problem. Moreover, a testing requirement reduction is proposed and makes EBSA more suitable into VS-CTDG than ever. Through benchmarks, EBSA is proved an effective approach.

Development of Regional Public Election Analytics Application In Indonesia Using Data from Twitter

Abstract: Many social media users use Twitter to comment on political issues especially during the elections. In order to extract information from Indonesian people's tweets about the election, this research made an analytics application to analyze the prospective regional election in Indonesia. The research method using Software Reuse, which relies on Reusable Component-Based Software Engineering (CBSE). Some steps done in this research are requirement specification, component analysis, requirement modification, system design with reuse, development and integration, and system validation. In system validation, we have done functional requirement testing in the form of black box testing and non-functional requirement test in the form of performance (running time), accuracy, and usability. The results of black box testing indicate that the functional system has been run according to the required specifications that have been made. The results of the classification testing using SVM method showed its average accuracy is 90.73% in general election mode and 84.58% in the election discussing each of the candidates. The average process time of backend module is 665 seconds from 1500 data tweets. In addition, the dashboard module usability result of the user's understanding of the application based on the questionnaire results shows an average value 2.7 of the value range 0-3 of all users who populated the questioner

Intelligent Fuzz Testing Framework for Finding Hidden Vulnerabilities in Automotive Environment

Abstract: Penetration Testing (pentesting) is most popular and effective method for finding vulnerabilities in the enterprise network and also in the automotive networks. It is used to find and patch loopholes in the systems. There are few pentesting methods based on knowledge of the system viz. white-box testing, black box testing and grey box testing. Among all these techniques Black-box testing is one of the most challenging attack scenario as there is a lack of information available about the system to an attacker as well as tester. To find out system loopholes adequately, pentesting is necessary. Fuzzing method is used to find the defects in the system i.e. unexpected behavior of the server. It is carried out by giving large invalid input data called as Fuzz. Fuzzing method can be applied for many interfaces on automotive systems like Bluetooth, Wi-Fi, Ethernet, USB and CAN. Many Fuzzing techniques are available and can be used to develop Fuzzer according to the requirements. Vehicle networks which work on Unified Diagnostic Service (UDS) protocol are vulnerable and hence vulnerabilities can be exploited by an attacker. Proposed solution introduces a framework to find the hidden vulnerabilities in automotive environment. A fuzzing tool which is specifically used for automotive systems is designed and developed. It uses mutation based approach for invalid input creation. Comparing relative merits of the existing systems is also a crucial task. In spite of having different protocol descriptions for the existing systems, proposed system can be compared to some extent but not completely with the existing systems.

CoMa: Development of Gamification-based E-learning

Abstract: There has been evidence that many of e-learning implementations fail to achieve their learning objectives due to incompatibility and lack of knowledge in developing an online information system. This failure could lead to frustration, confusion, and a decrease in the student engagement in learning. Moreover, transferring physical materials to the digital ones could result in boredom because there is no interactivity in the learning process. As a consequence, student engagement in learning decreases. Gamification concept may be used to increase student engagement in learning. This study develops an e-learning system by adding gamification concept. Gamification elements used in this study are leaderboard, activity point, experience point, badge, challenges, leveling, and add friends. The e-learning system is implemented in the Java Programming course. This e-learning is called Code Mania (CoMa). The e-learning is a web-based system that allows the user to interact easily. The e-learning also has an automatic evaluation feature to evaluate student work. The developed learning system is evaluated using whitebox testing that is path testing and blackbox testing. The result from the testing shows that the e-learning system could run well as specified in the system requirement.

Implementing Forward, Backward Chaining and Certainty Factor in Responsive Web-Based Expert System of Cow Disease

Abstract: This research aims to design and build expert systems of cow disease to assist farmers in identifying cattle diseases. A large number of cattle in Banyumas is not matched by the number of veterinarians, the Department of Fisheries and Livestock (DINKANAK). Banyumas records 961 cases of sick cows in 2016. This expert system is expected to assist cattle ranchers to identify cow disease and symptom-based remedies illness-symptoms. By using Inference Forward and Backward chaining which is a search method or tracking technique by using information from breeders, and Certainty Factor is used to accommodate the uncertainty of thinking of a data expert that is Doctor the process of extracting knowledge by interview. In this research system development using ESDLC (Expert System Development Life Cycle) with stages of Planning, Knowledge Acquisition, Implementation, and Evaluation. Testing is done with two approaches are Alpha Testing and Beta Testing. Alpha Testing conducted on the developer side to test the functional system using Black Box Testing method result all functional system can function well. Beta Testing is aimed at user acceptance by a Questionnaire method yields an average score of 76% or usability and the quality of system information is easy to understand.

White box testing

Abstract: The source code of a software artifact may be scanned, and a call tree model with leaf nodes may be generated based on the scan. A set of test cases can be executed against the software artifact and log data from the execution can be collected. A set of untested leaf nodes can be detected and a new set of test cases can be generated to test the untested nodes. The new set of test cases are executed and a subset of the test cases which cover the previously untested nodes are added to the existing set of test cases.

On Automated Co-Simulation Testing of Functional Requirements for Distributed Substation Automation Systems

Abstract: The smart grid is positioned as the next generation energy distribution system which incorporates distributed renewable energy and digital communication infrastructure. The Smart Grid is considered by many as a complex Cyber-Physical System, a system with tight integration between its modular cyber and physical processes interacting in a networked environment. Traditionally, the method of testing Smart Grid automation systems is based on device-based testing and this is largely influenced by the bottom-up design methods that are used widely in the engineering of the substation automation system. However, with the advent of the Smart Grid and the introduction of distributed based automation systems, requirement-driven testing of the whole automation system may be more beneficial than device based testing. In this paper, a scripted co-simulation platform is introduced with the purpose of automating the testing of functional requirements in distributed substation automation systems. The scripted co-simulation testing framework is illustrated on a CIGRE case study where the distributed automation system is validated against the functional requirements.