Showing papers by "Sebastian Elbaum published in 2005"

Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact

Abstract: Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has had and can be expected to have.

Leveraging user-session data to support Web application testing

Abstract: Web applications are vital components of the global information infrastructure, and it is important to ensure their dependability. Many techniques and tools for validating Web applications have been created, but few of these have addressed the need to test Web application functionality and none have attempted to leverage data gathered in the operation of Web applications to assist with testing. In this paper, we present several techniques for using user session data gathered as users operate Web applications to help test those applications from a functional standpoint. We report results of an experiment comparing these new techniques to existing white-box techniques for creating test cases for Web applications, assessing both the adequacy of the generated test cases and their ability to detect faults on a point-of-sale Web application. Our results show that user session data can be used to produce test suites more effective overall than those produced by the white-box techniques considered; however, the faults detected by the two classes of techniques differ, suggesting that the techniques are complementary.

Profiling deployed software: assessing strategies and testing opportunities

Abstract: An understanding of how software is employed in the field can yield many opportunities for quality improvements. Profiling released software can provide such an understanding. However, profiling released software is difficult due to the potentially large number of deployed sites that must be profiled, the transparency requirements at a user's site, and the remote data collection and deployment management process. Researchers have recently proposed various approaches to tap into the opportunities offered by profiling deployed systems and overcome those challenges. Initial studies have illustrated the application of these approaches and have shown their feasibility. Still, the proposed approaches, and the tradeoffs between overhead, accuracy, and potential benefits for the testing activity have been barely quantified. This paper aims to overcome those limitations. Our analysis of 1,200 user sessions on a 155 KLOC deployed system substantiates the ability of field data to support test suite improvements, assesses the efficiency of profiling techniques for released software, and the effectiveness of testing efforts that leverage profiled field data.

Helping end-users "engineer" dependable Web applications

Abstract: End-user programmers are increasingly relying on Web authoring environments to create Web applications. Although often consisting primarily of Web pages, such applications are increasingly going further, harnessing the content available on the Web through "programs" that query other Web applications for information to drive other tasks. Unfortunately, errors can be pervasive in Web applications, impacting their dependability. This paper reports the results of an exploratory study of end-user Web application developers, performed with the aim of exposing prevalent classes of errors. The results suggest that end-users struggle the most with the identification and manipulation of variables when structuring requests to obtain data from other Web sites. To address this problem, we present a family of techniques that help end user programmers perform this task, reducing possible sources of error. The techniques focus on simplification and characterization of the data that end-users must analyze while developing their Web applications. We report the results of an empirical study in which these techniques are applied to several popular Web sites. Our results reveal several potential benefits for end-users who wish to "engineer" dependable Web applications

The first workshop on end-user software engineering

Abstract: WEUSE is a workshop dedicated to the problems faced by end-user programmers, and research that can address those problems.

Profiling Deployed Software: Strategic Probe Placement

Abstract: Profiling deployed software provides valuable insights for quality improvement activities. The probes required for profiling, however, can cause an unacceptable performance overhead for users. In previous work we have shown that significant overhead reduction can be achieved, with limited information loss, through the distribution of probes across deployed instances. However, existing strategies for probe distribution do not account for several relevant factors: acceptable overheads may vary, the distributions to be deployed may be limited, profiled events may have different likelihoods, and the user pool composition may be unknown. This paper evaluates strategies for probe distribution while manipulating these factors through an empirical study. Our findings indicate that for tight overhead bounds: 1) deploying numerous versions with complementary probe distributions can compensate for the reduction of probes per deployed instance, and 2) techniques that balance probe allocation, consider groupings and prior-allocations, can generate distributions that retain significantly more field information.

Garbage collection in the presence of remote objects: an empirical study

Abstract: Most virtual machine implementations employ generational garbage collection to manage dynamically allocated memory. Studies have shown that these generational schemes work efficiently in desktop-like applications where most objects are short-lived. The performance of generational collectors, however, has been rarely studied in the context of distributed systems. Given the increasing popularity of such systems, and the distinct type of objects they introduce to support the distributed paradigm, providing insights into their memory allocation behavior could have a large impact on the design of future garbage collection techniques, and in the implementation of such distributed systems as well. This work presents one of the first attempts to characterize the lifespan of objects in distributed systems. First, we empirically study the differences in lifespan of remote and local objects. Second, we investigate the effects of ephemeral heap size and workload on the lifespan or remote objects. Last, we utilize the insights gained through the experiment to improve the efficiency of a generational collection scheme through the segregation of objects based on their locality.

Fault patterns in Matlab

Abstract: Fault patterns are code idioms that may constitute faults. Software engineers have various program analysis techniques and tools to assist them in the detection of such patterns, resulting in increased software quality. End user programmers, however, often lack such support. In this paper we take a first step to address this limitation in the context of Matlab. First, we adapt fault patterns commonly used in other programming languages to Matlab. Second, we present a tool to detect such patterns in fifteen popular Matlab programs. Our results reveal that these simple and quickly identifiable patterns are commonly found in Matlab programs developed by end users and shared across the large Matlab community of end user programmers.