A runtime adaptable search processor is disclosed. The search processor provides high speed content search capability to meet the performance need of network line rates growing to 1Gbps, IOGbps and higher. The search processor provides a unique combination of NFA and DFA based search engines that can processincoming data in parallel to perform the search against the specific rules programmed in the search engines. The procesor architecture also provides capabilities to transport and process Internet Protocol (IP) packets from Layer 2 through transport protocol layer and may also provide packet inspection through Layer 7. A security system is also disclosed that enables a new way of implementing security capabilities inside enterprise networks in a distributed manner using a protocol processing hardware with appropriate security features.

Runtime adaptable search processor

There is a growing demand for network devices capable of examining the content of data packets in order to improve network security and provide application-specific services. Most high performance systems that perform deep packet inspection implement simple string matching algorithms to match packets against a large, but finite set of strings. owever, there is growing interest in the use of regular expression-based pattern matching, since regular expressions offer superior expressive power and flexibility. Deterministic finite automata (DFA) representations are typically used to implement regular expressions. However, DFA representations of regular expression sets arising in network applications require large amounts of memory, limiting their practical application.In this paper, we introduce a new representation for regular expressions, called the Delayed Input DFA (D2FA), which substantially reduces space equirements as compared to a DFA. A D2FA is constructed by transforming a DFA via incrementally replacing several transitions of the automaton with a single default transition. Our approach dramatically reduces the number of distinct transitions between states. For a collection of regular expressions drawn from current commercial and academic systems, a D2FA representation reduces transitions by more than 95%. Given the substantially reduced space equirements, we describe an efficient architecture that can perform deep packet inspection at multi-gigabit rates. Our architecture uses multiple on-chip memories in such a way that each remains uniformly occupied and accessed over a short duration, thus effectively distributing the load and enabling high throughput. Our architecture can provide ostffective packet content scanning at OC-192 rates with memory requirements that are consistent with current ASIC technology.

/pdf/algorithms-to-accelerate-multiple-regular-expressions-3oxorq3phy.pdf

Algorithms to accelerate multiple regular expressions matching for deep packet inspection

Packet content scanning at high speed has become extremely important due to its applications in network security, network monitoring, HTTP load balancing, etc. In content scanning, the packet payload is compared against a set of patterns specified as regular expressions. In this paper, we first show that memory requirements using traditional methods are prohibitively high for many patterns used in packet scanning applications. We then propose regular expression rewrite techniques that can effectively reduce memory usage. Further, we develop a grouping scheme that can strategically compile a set of regular expressions into several engines, resulting in remarkable improvement of regular expression matching speed without much increase in memory usage. We implement a new DFA-based packet scanner using the above techniques. Our experimental results using real-world traffic and patterns show that our implementation achieves a factor of 12 to 42 performance improvement over a commonly used DFA- based scanner. Compared to the state-of-art NFA-based implementation, our DFA-based packet scanner achieves 50 to 700 times speedup.

/pdf/fast-and-memory-efficient-regular-expression-matching-for-1072pu59iz.pdf

Fast and memory-efficient regular expression matching for deep packet inspection

Reconfigurable computing is becoming increasingly attractive for many applications. This survey covers two aspects of reconfigurable computing: architectures and design methods. The paper includes recent advances in reconfigurable architectures, such as the Alters Stratix II and Xilinx Virtex 4 FPGA devices. The authors identify major trends in general-purpose and special-purpose design methods. It is shown that reconfigurable computing designs are capable of achieving up to 500 times speedup and 70% energy savings over microprocessor implementations for specific applications.

/pdf/reconfigurable-computing-architectures-and-design-methods-2g6ogsz88l.pdf

Reconfigurable computing: architectures and design methods

Tools and Algorithms for the Construction and Analysis of Systems, 7th International Conference, TACAS 2001, Genova, Italy, April 2-6, 2001

In this paper, we present a scalable FPGA design methodology for searching network packet payloads for a large number of patterns, including complex regular expressions. The efficiency of the technique enables a current-generation FPGA device to support pattern-matching at network rates from 1 Gbps to 100 Gbps and beyond. It offers flexible trade-offs between character capacity, throughput, and data bus width and rate. This allows the approach to be used in a wide range of devices from low-end home network appliances to high-end backbone routers. Suitable network applications for the FPGA pattern-matcher include firewalls, network intrusion detection, email virus scanning, and junk-email identification. In this work, we use a standard set of patterns from an intrusion detection system to demonstrate the performance and scalability of our design with a real-world application.

http://cial.csie.ncku.edu.tw/st2007/pdf/Scalable%20Pattern%20Matching%20for%20High%20Speed%20Networks.pdf

Scalable pattern matching for high speed networks

This paper presents techniques for designing pattern matching circuits for complex regular expressions, such as those found in network intrusion detection patterns. We have developed a pattern-matching co-processor that supports all the pattern matching functions of the Snort rule language [3]. In order to achieve maximum pattern capacity and throughput, the design focuses on minimizing circuit area while maintaining high clock speed. Using our approach, we are able to store the entire current Snort rule database consisting of over 1,500 rules and 17,000 characters into a single one-million-gate FPGA while comparing all patterns against traffic at gigabit rates.

Efficient reconfigurable logic circuits for matching complex network intrusion detection patterns

The need for building high-speed NIDS that can reliably generate alerts as intrusions occur and have the intrinsic ability to scale as network infrastructure and attack sophistication evolves has been discussed in this chapter. The key design principles are analyzed and it has been argued that network intrusion-detection functions should be carried out by distributed and collaborative NNIDS at the end hosts. It is shown that an NNIDS running on the network interface instead of the host operating system can provide increased protection, reduced vulnerability to circumvention, and much lower overhead. The chapter also describes the experience in implementing a prototype NNIDS, based on Snort, an Intel IXP 1200, and a Xilinx Virtex-1000 FPGA. These experiments help to identify the performance bottlenecks and give insights on how to improve the design. System stress tests shows that the embedded NNIDS can handle high-speed traffic without packet drops and achieve the same performance as the Snort software running on a dedicated high-end computer system. Ongoing work includes optimizing the performance of NNIDS, developing strategies for sustainable operation of the NNIDS under attacks through adaptation and active countermeasures, studying algorithms for distributed and collaborative intrusion detection, and further developing the analytical models for buffer and processor allocation. Also tested were FPGA pattern-matching designs that approach 10 Gbps throughput with the entire Snort ruleset using a Xilinx Virtex2 device. A better understanding of the design principles and implementation techniques for building high-speed has been provided, along with reliable, and scalable network intrusion detection systems.

/pdf/a-hardware-platform-for-network-intrusion-detection-and-3e9ydbcfvt.pdf

A hardware platform for network intrusion detection and prevention

There has been a significant volume of recent work on FPGA designs for pattern matching. Although various pattern-matching architectures have been presented, attempts to compare different designs have been inconclusive, or even misleading, due to the lack of a common evaluation framework. In this paper, we present an analytical model of FPGA pattern-matching architectures that quantitatively expresses the relationships between pattern properties, circuit area, and circuit delay. We derive equations that show how the performance of each architecture is dependent on the properties of the pattern set. This model enables many different pattern-matching architectures to be compared in order to determine the optimal design for a given pattern-matching application.

/pdf/a-unified-model-of-pattern-matching-circuit-architectures-epc46hfpk4.pdf

A Unified Model of Pattern-Matching Circuit Architectures

This paper explores the design and analysis of an FPGA module that implements pattern-matching functionality for the network intrusion detection problem. The specific features of the pattern-matcher include support for complex regular expressions and approximate matching with bounded substitutions, insertions, and deletions. A module generator is presented that utilizes non-deterministic finite automata to dynamically create efficient circuits for matching patterns specified with a standard rule language. The logic complexity and performance of the generated circuits is measured and analyzed. Results indicate our techniques yield circuits that are more than twice as dense as other reported designs, while maintaining the throughput necessary for processing at gigabit line speeds and beyond. The FPGA pattern-matching processor is integrated with other hardware and software components to form a complete network intrusion detection system.

Christopher R. Clark

Papers

Scalable pattern matching for high speed networks

Efficient reconfigurable logic circuits for matching complex network intrusion detection patterns

A hardware platform for network intrusion detection and prevention

A Unified Model of Pattern-Matching Circuit Architectures

A pattern-matching co-processor for network intrusion detection systems