According to one embodiment, a computer-implemented method includes accessing, using one or more processing units, a first file of a plurality of files requested to be analyzed for malware. Each of the plurality of files corresponds to a respective remote client of a plurality of remote clients. Further, the method includes: processing, using the one or more processing units, an analysis of the first file for malware; and generating an output comprising an indication of whether the first file comprises malware. The method also includes accessing, using the one or more processing units, an address for a first remote client of the plurality of remote clients. The first remote client is the respective remote client corresponding to the first file. In addition, the method includes: sending, using the one or more processing units, the output in a communication addressed to the first remote client corresponding to the first file.

Distributed malware detection

Internet traffic classification has been the subject of intensive study since the birth of the Internet itself. Indeed, the evolution of approaches for traffic classification can be associated with the evolution of the Internet itself and with the adoption of new services and the emergence of novel applications and communication paradigms. Throughout the years many approaches have been proposed for addressing technical issues imposed by such novel services. Deep-Packet Inspection (DPI) has been a very important research topic within the traffic classification field and its concept consists of the analysis of the contents of the captured packets in order to accurately and timely discriminate the traffic generated by different Internet protocols. DPI was devised as a means to address several issues associated with port-based and statistical-based classification approaches in order to achieve an accurate and timely traffic classification. Many research works proposed different DPI schemes while many open-source modules have also become available for deployment. Surveys become then valuable tools for performing an overall analysis, study and comparison between the several proposed methods. In this paper we present a survey in which a complete and thorough analysis of the most important open-source DPI modules is performed. Such analysis comprises an evaluation of the classification accuracy, through a common set of traffic traces with ground truth, and of the computational requirements. In this manner, this survey presents a technical assessment of DPI modules and the analysis of the obtained evaluation results enable the proposal of general guidelines for the design and implementation of more adequate DPI modules.

A Survey of Payload-Based Traffic Classification Approaches

Deep packet inspection (DPI) is widely used in content-aware network applications such as network intrusion detection systems, traffic billing, load balancing, and government surveillance. Pattern matching is a core and critical step in DPI, which checks the payload of each packet for known signatures (patterns) in order to identify packets with certain characteristics (e.g., malicious packets that carry viruses or worms). Regular expression is the major tool for signature description due to its powerful and flexible expressive ability. However, this flexibility also brings great challenges for efficient implementation in practice. Despite of hundreds to thousands of empirical proposals, wire-speed matching for large scale regular expressions still remains a big challenge. The gap between the matching throughput and the link speed is widening with the ever-increasing network link speed and pattern scale. This survey begins with a full-scale application background of DPI and technical background of regular expression matching in order to provide a global view and essential knowledge for readers. We then analyze the challenges in regular expression matching originated from the state explosion of finite state automaton used for regular expression matching. The nature of state explosion is analyzed in details, and the state-of-the-art solutions are grouped into categories of methods to relieve state expansion and methods to avoid state explosion, suggestions are also provided for building compact and efficient automata in different scenarios. Furthermore, proposals employing parallel platforms, including field-programmable gate array, GPU, general multi-processors, and ternary content addressable memory, to accelerate the matching process are introduced and thoroughly discussed. We also provide guidelines for efficient deployment for each of these platforms.

A Survey on Regular Expression Matching for Deep Packet Inspection: Applications, Algorithms, and Hardware Platforms

Detecting attacks disguised by evasion techniques is a challenge for signature-based Intrusion Detection Systems (IDSs) and Intrusion Prevention Systems (IPSs) This study examines five common evasion techniques to determine their ability to evade recent systems The denial-of-service (DoS) attack attempts to disable a system by exhausting its resources Packet splitting tries to chop data into small packets, so that a system may not completely reassemble the packets for signature matching Duplicate insertion can mislead a system if the system and the target host discard different TCP/IP packets with a duplicate offset or sequence Payload mutation fools a system with a mutative payload Shellcode mutation transforms an attacker's shellcode to escape signature detection This study assesses the effectiveness of these techniques on three recent signature-based systems, and among them, explains why Snort can be evaded The results indicate that duplicate insertion becomes less effective on recent systems, but packet splitting, payload mutation and shellcode mutation can be still effective against them

/pdf/evasion-techniques-sneaking-through-your-intrusion-detection-56amt8zd2r.pdf

Evasion Techniques: Sneaking through Your Intrusion Detection/Prevention Systems

We present the design and implementation of a novel anti-malware system called SplitScreen. SplitScreen performs an additional screening step prior to the signature matching phase found in existing approaches. The screening step filters out most non-infected files (90%) and also identifies malware signatures that are not of interest (99%). The screening step significantly improves end-to-end performance because safe files are quickly identified and are not processed further, and malware files can subsequently be scanned using only the signatures that are necessary. Our approach naturally leads to a network-based anti-malware solution in which clients only receive signatures they needed, not every malware signature ever created as with current approaches. We have implemented SplitScreen as an extension to ClamAV, the most popular open source anti-malware software. For the current number of signatures, our implementation is 2x faster and requires 2x less memory than the original ClamAV. These gaps widen as the number of signatures grows.

/pdf/splitscreen-enabling-efficient-distributed-malware-detection-1c0k822wub.pdf

SplitScreen: Enabling efficient, distributed malware detection

The efficiency of string matching algorithms is essential for network content security applications, such as intrusion detection systems, anti-virus systems, and Web content filters. This work reviews typical algorithms and profiles their performance under various situations to study the influence of the number, the length, and the character distribution of the signatures on performance. This profiling can reveal the most efficient algorithm in each situation. A fast verification method for some string matching algorithms is also proposed. This work then analyzes the signature characteristics of three content security applications and replaces their original algorithms with the most efficient ones in the profiling. The improvement for both real and synthetic sample data is observed. For example, an open source anti-virus package, ClamAV, is five times faster after the revision. This work features comprehensive profiling results of typical string matching algorithms and observations of their application on network content security. The results can enlighten the choice of a proper algorithm in practical design.

/pdf/profiling-and-accelerating-string-matching-algorithms-in-14kl2yowk8.pdf

Zhi-Xiang Li

Papers

Profiling and accelerating string matching algorithms in three network content security applications