Source code querying tools allow programmers to explore relations between different parts of the code base. This paper describes such a tool, named CodeQuest. It combines two previous proposals, namely the use of logic programming and database systems. As the query language we use safe Datalog, which was originally introduced in the theory of databases. That provides just the right level of expressiveness; in particular recursion is indispensable for source code queries. Safe Datalog is like Prolog, but all queries are guaranteed to terminate, and there is no need for extra-logical annotations. Our implementation of Datalog maps queries to a relational database system. We are thus able to capitalise on the query optimiser provided by such a system. For recursive queries we implement our own optimisations in the translation from Datalog to SQL. Experiments confirm that this strategy yields an efficient, scalable code querying system.

Code Quest : Scalable Source Code Queries with Datalog

Searching code samples in a code repository is an important part of program comprehension. Most of the existing tools for code search support syntactic element search and regular expression pattern search. However, they are text-based and hence cannot handle queries which are syntactic patterns. The proposed solutions for querying syntactic patterns using specialized query languages present a steep learning curve for users. The querying would be more user-friendly if the syntactic pattern can be formulated in the underlying programming language (as a sample code snippet) instead of a specialized query language. In this paper, we propose a solution for the query by example problem using Abstract Syntax Tree (AST) structural similarity match. The query snippet is converted to an AST, then its subtrees are compared against AST subtrees of source files in the repository and the similarity values of matching subtrees are aggregated to arrive at a relevance score for each of the source files. To scale this approach to large code repositories, we use locality-sensitive hash functions and numerical vector approximation of trees. Our experimental evaluation involves running control queries against a real project. The results show that our algorithm can achieve high precision (0.73) and recall (0.81) and scale to large code repositories without compromising quality.

Query by example in large-scale code repositories

Static analysis is an efficient approach for software assurance. It is indicated that its most effective usage is to perform analysis in an interactive way through the software development process, which has a high performance requirement. This paper concentrates on rule-based static analysis tools and proposes an optimized rule-checking algorithm. Our technique improves the performance of static analysis tools by filtering vulnerability rules in terms of characteristic objects before checking source files. Since a source file always contains vulnerabilities of a small part of rules rather than all, our approach may achieve better performance. To investigate our technique’s feasibility and effectiveness, we implemented it in an open source static analysis tool called PMD and used it to conduct experiments. Experimental results show that our approach can obtain an average performance promotion of 28.7% compared with the original PMD. While our approach is effective and precise in detecting vulnerabilities, there is no side effect.

Efficient vulnerability detection based on an optimized rule-checking static analysis technique

A system and method for code virtualization and remote process call code on a user device. The method includes selecting a remote process on a remote server, the remote process correlated to at least one remote service, and receiving from at least one virtual code provider (VCP) on the remote server, a code snippet with parameters from pre-defined fields to call the remote process. The method installs a software development kit (SDK) downloaded from the remote server, onto the user device. The method inserts the code snippet based on the SDK into a local code base on the user device, and inserts parameter values into the code snippet and deploying the code snippet such that the code snippet calls the remote process with the SDK from the user device.

Method and apparatus for code virtualization and remote process call generation

The first generation of software retrieval systems developed some 25 years ago used simple bibliographic indexing techniques adapted from library science to support the retrieval of relatively small numbers of in-house software artifacts. While these were sufficient at the time, they were completely unscaleable to the vast numbers of software artifacts available today. The second generation of software search engines, representing the state-of-the-practice today, tackles this problem by using full-text search frameworks such as Lucene to support text-based searches on large software collections. However, these typically provide no inherent support for sophisticated search use cases which exploit the structure and “meaning” of software artifacts. In this chapter we describe the core techniques used in current text-based code search engines and advanced techniques that can be used to support sophisticated forms of searches that exploit the structure of software. We then survey the challenges and opportunities encountered in the development of the next (third) generation of software search engines based on new, currently emerging data storage platforms.

Artifact Representation Techniques for Large-Scale Software Search Engines

While analyzing a log file of a text-based source code search engine we discovered that developers search for fine-grained syntactical patterns in 36\% of queries. Currently, to cope with queries of this kind developers need to use regular expressions, to add redundant terms to the query or to combine searching with other tools provided by the development environment. To improve the expressiveness of the queries, these can be formulated as tree patterns of abstract syntax trees. These search patterns can be expressed by using query languages, such as XPath. However, developers usually do not work with either XPath or with AST. To shield developers from the complexity of query formulation we propose using sample code snippets as queries. The novelty of our approach is the combination of a query language that is very close to the surface programming language and a special database technology to store a large amount of abstract syntax trees. The advantage of this approach over existing source code query languages and search engines is the performance of both query formulation and query execution. This paper describes the technical details of the method and illustrates the value of this approach with performance measures and an industrial controlled experiment. All developers were able to complete the tasks of the experiment faster and more accurately by using our tool (ACS) than by using a text-based search engine. The number of false positives in the result lists was significantly decreased.

Precise and Scalable Querying of Syntactical Source Code Patterns Using Sample Code Snippets and a Database

One of the challenges that Web service providers face is service evolution management. In general, the challenge is to ensure the substitutability of service versions, i.e. correct functioning of all ongoing client applications relying on the old version of a service after the version has been substituted with a new one. Unfortunately, no currently available design approach can guarantee a perfectly extensible architecture that preserves full backward compatibility during its evolution. Hence, incompatibilities are very likely to occur if an old version of a service is replaced with a new one. This paper addresses the incompatibility problem and describes a solution to the problem. This solution is based upon the already known design pattern of message translation and the ASP.NET 2.0 Web service platform. Using the platform’s API the standard ASP.NET pipeline has been augmented with an additional step of applying XSL transformations to the XML payload of the messages. The solution is then verified against the Electronic Commerce Service from Amazon.com web services suite. Thus, the contribution of the work is a new .NET implementation of the translator pattern.

Jan Karstens

Papers

Precise and Scalable Querying of Syntactical Source Code Patterns Using Sample Code Snippets and a Database

Resolving incompatibility during the evolution of web services with message conversion