Showing papers on "Test harness published in 2020"

HPC-MixPBench: An HPC Benchmark Suite for Mixed-Precision Analysis

Abstract: With the increasing interest in applying approximate computing to HPC applications, representative benchmarks are needed to evaluate and compare various approximate computing algorithms and programming frameworks. To this end, we propose HPC-MixPBench, a benchmark suite consisting of a representative set of kernels and benchmarks that are widely used in HPC domain. HPC-MixPBench has a test harness framework where different tools can be plugged in and evaluated on the set of benchmarks. We demonstrate the effectiveness of our benchmark suite by evaluating several mixed-precision algorithms implemented in FloatSmith, a tool for floating-point mixed-precision approximation analysis. We report several insights about the mixed-precision algorithms that we compare, which we expect can help users of these methods choose the right method for their workload. We envision that this benchmark suite will evolve into a standard set of HPC benchmarks for comparing different approximate computing techniques.

The LDBC Graphalytics Benchmark.

Abstract: In this document, we describe LDBC Graphalytics, an industrial-grade benchmark for graph analysis platforms. The main goal of Graphalytics is to enable the fair and objective comparison of graph analysis platforms. Due to the diversity of bottlenecks and performance issues such platforms need to address, Graphalytics consists of a set of selected deterministic algorithms for full-graph analysis, standard graph datasets, synthetic dataset generators, and reference output for validation purposes. Its test harness produces deep metrics that quantify multiple kinds of systems scalability, weak and strong, and robustness, such as failures and performance variability. The benchmark also balances comprehensiveness with runtime necessary to obtain the deep metrics. The benchmark comes with open-source software for generating performance data, for validating algorithm results, for monitoring and sharing performance data, and for obtaining the final benchmark result as a standard performance report.

Realization of a state machine based detection for Track Segments in the Trigger System of the Belle II Experiment

Abstract: The Belle II experiment relies on an online level 1 trigger system to reduce the background and achieve the targeted frequency of 30 kHz. Here the basis for all trigger decisions based on data from the Central Drift Camber is the track segment finding. To improve both efficiency and maintainability we restructured the original combinatorial approach to finite state machines. The new implementation is saving about 20% of FPGA slices. To achieve high test coverage an automated test framework was developed for design time validation. Operational correctness is achieved by integration in cosmic ray tests.

A Performance Comparison of WireGuard and OpenVPN

Abstract: A fundamental problem that confronts virtual private network (VPN) applications is the overhead on throughput, ease of deployment and use, and overall utilization. WireGuard is a recently introduced light and secure cross-platform VPN application. It aims to simplify the process of setting up a secure connection while utilizing the multi-threading capability and minimizing the use of bandwidth. There have been several follow-up studies on WireGuard since its birth, most of which focus on the security analysis of the protocol. Despite the author's claim that WireGuard has impressive wins over OpenVPN and IPsec, there is no rigorous analysis of its performance to date. This paper presents a performance comparison of WireGuard and its main rival OpenVPN on various metrics. We construct an automated test framework and deploy it on a total of eight nodes, including remote AWS instances and local virtual machines. Our test results clearly show two main edges that WireGuard has over OpenVPN, its performance on multi-core machines and its light codebase.

Revisiting Test Impact Analysis in Continuous Testing From the Perspective of Code Dependencies

Abstract: In continuous testing, developers execute automated test cases once or even several times per day to ensure the quality of the integrated code. Although continuous testing helps ensure the quality of the code and reduces maintenance effort, it also significantly increases test execution overhead. In this thesis, we empirically evaluate the effectiveness of test impact analysis from the perspective of code dependencies in the continuous testing setting. We first applied test impact analysis to one year of software development history in 11 large-scale open-source systems. We found that even though the number of changed files is small in daily commits (median ranges from 3 to 28 files), around 50% or more of the test cases are still impacted and need to be executed. Motivated by our finding, we further studied the code dependencies between source code files and test cases, and among test cases. We found that 1) test cases often focus on testing the integrated behaviour of the systems; 2) 18% of the test cases have dependencies with other test cases, and test case inheritance is the most common cause of test case dependencies; 3) We documented four dependency-related test smells that we uncovered in our manual study. Our study provides the first step towards studying and understanding the effectiveness of test impact analysis in the continuous testing setting and provides insights on improving test design and execution.

Practical Automatic Lightweight Nondeterminism and Flaky Test Detection and Debugging for Python

Abstract: A critically important, but surprisingly neglected, aspect of system reliability is system predictability. Many soft-ware systems are implemented using mechanisms (unsafe languages, concurrency, caching, stochastic algorithms, environmental dependencies) that can introduce unexpected and unwanted behavioral nondeterminism. Such nondeterministic behavior can result in software bugs and flaky tests as well as causing problems for test reduction, differential testing, and automated regression test generation. We show that lightweight techniques, requiring little effort on the part of developers, can extend an existing testing system to allow detection and debugging of nondeterminism. We show how to make delta-debugging effective for probabilistic faults in general, and that our methods can improve mutation score by 6% for a strong, full differential test harness for a widely used mock file system.

DCO analyzer: local controllability and observability analysis and enforcement of distributed test scenarios

Abstract: To ensure interoperability and the correct behavior of heterogeneous distributed systems in key scenarios, it is important to conduct automated integration tests, based on distributed test components (called local testers) that are deployed close to the system components to simulate inputs from the environment and monitor the interactions with the environment and other system components. We say that a distributed test scenario is locally controllable and locally observable if test inputs can be decided locally and conformance errors can be detected locally by the local testers, without the need for exchanging coordination messages between the test components during test execution (which may reduce the responsiveness and fault detection capability of the test harness). DCO Analyzer is the first tool that checks if distributed test scenarios specified by means of UML sequence diagrams exhibit those properties, and automatically determines a minimum number of coordination messages to enforce them. The demo video for DCO Analyzer can be found at https://youtu.be/LVIusK36_bs.

An Automated Test Framework for Regression Test Selection for Web Services

Abstract: Web services represent a software paradigm that is based on service-oriented architecture. They represent Open Internet Standards and are used to represent business functionalities on the Internet. Due to changing business scenarios, these services are frequently evolving. To ensure proper quality and maintaining them becomes a tedious task due to their dynamic nature. Software maintenance is an expensive activity that consumes a considerable amount of manpower and no testing hours. To reduce the maintenance cost is a need. Regression testing is a part of maintenance that is done every time the software is changed to ensure the correctness of the software. It consumes a considerable amount of maintenance costs. Thus it is a need to reduce regression testing. Regression test selection is a promising approach in order to reduce regression test cases thereby saving a considerable amount of time and effort. Thus in this paper, we have given a regression test selection framework that is automated for web-based applications like web services.

Voice controlled keyword generation for automated test framework

Abstract: Methods, systems and computer program products for voice controlled generation of keywords for test cases are provided. Aspects include receiving a verbal test plan for a computing system and analyzing the verbal test plan to identify a plurality of keywords. Aspects also include mapping each known keyword of the plurality of keywords to an existing grammar and obtaining, from a user, a definition for a new grammar for each unknown keyword of the plurality of keywords. Aspects further include analyzing a software being tested on the computing system and creating a test plan by respectively applying the new grammar and the existing grammar of the plurality of keywords to the software being tested.

Web-based application platform applying lean production methods to system delivery testing

Abstract: Implementations include receiving, by a DDT platform, computer-readable files including test data, the test data including data to execute at least one transaction during testing of a software system that is at least partially hosted by a vendor back-end system, the software system being configured for customer use, providing, by the DDT platform, one or more test scenarios for execution by the software system on the vendor back-end system, the one or more test scenarios including a set of activities to conduct transactions by the software system using at least a portion of the test data, scheduling, by a scheduler of the DDT platform, execution of the one or more test scenarios using a test harness of the DDT platform, receiving, by the DDT platform, test results from the vendor back-end system, and comparing, by the DDT platform, the test results to expected results to provide a comparison as output.

DCO Analyzer: Local Controllability and Observability Analysis and Enforcement of Distributed Test Scenarios.

Abstract: To ensure interoperability and the correct behavior of heterogeneous distributed systems in key scenarios, it is important to conduct automated integration tests, based on distributed test components (called local testers) that are deployed close to the system components to simulate inputs from the environment and monitor the interactions with the environment and other system components. We say that a distributed test scenario is locally controllable and locally observable if test inputs can be decided locally and conformance errors can be detected locally by the local testers, without the need for exchanging coordination messages between the test components during test execution (which may reduce the responsiveness and fault detection capability of the test harness). DCO Analyzer is the first tool that checks if distributed test scenarios specified by means of UML sequence diagrams exhibit those properties, and automatically determines a minimum number of coordination messages to enforce them.