Deep packet inspection is playing an increasingly important role in the design of novel network services. Regular expressions are the language of choice for writing signatures, but standard DFA or NFA representations are unsuitable for high-speed environments, requiring too much memory, too much time, or too much per-flow state. DFAs are fast and can be readily combined, but doing so often leads to state-space explosion. NFAs, while small, require large per-flow state and are slow.We propose a solution that simultaneously addresses all these problems. We start with a first-principles characterization of state-space explosion and give conditions that eliminate it when satisfied. We show how auxiliary variables can be used to transform automata so that they satisfy these conditions, which we codify in a formal model that augments DFAs with auxiliary variables and simple instructions for manipulating them. Building on this model, we present techniques, inspired by principles used in compiler optimization, that systematically reduce runtime and per-flow state. In our experiments, signature sets from Snort and Cisco Systems achieve state-space reductions of over four orders of magnitude, per-flow state reductions of up to a factor of six, and runtimes that approach DFAs.

/pdf/deflating-the-big-bang-fast-and-scalable-deep-packet-2jbh0hyw7j.pdf

Deflating the big bang: fast and scalable deep packet inspection with extended finite automata

Automata-based representations and related algorithms have been applied to address several problems in information security, and often the automata had to be augmented with additional information. For example, extended finite-state automata (EFSA) augment finite- state automata (FSA) with variables to track dependencies between arguments of system calls. In this paper, we introduce extended finite automata (XFAs) which augment FSAs with finite scratch memory and instructions to manipulate this memory. Our primary motivation for introducing XFAs is signature matching in Network Intrusion Detection Systems (NIDS). Representing NIDS signatures as deterministic finite-state automata (DFAs) results in very fast signature matching but for several classes of signatures DFAs can blowup in space. Using nondeterministic finite-state automata (NFA) to represent NIDS signatures results in a succinct representation but at the expense of higher time complexity for signature matching. In other words, DFAs are time-efficient but space-inefficient, and NFAs are space- efficient but time-inefficient. In our experiments we have noticed that for a large class of NIDS signatures XFAs have time complexity similar to DFAs and space complexity similar to NFAs. For our test set, XFAs use 10 times less memory than a DFA-based solution, yet achieve 20 times higher matching speeds.

/pdf/xfa-faster-signature-matching-with-extended-automata-562hdrwnyb.pdf

XFA: Faster Signature Matching with Extended Automata

分析了基于处理器硬件虚拟化技术实现的KVM子系统的架构。针对KVM跟踪独立事件信息的局限性,提出一种新的KVM事件跟踪机制(kvmtrace)来达到性能调节的目的,并使用relayfs接口进行了设计与实现。同时探讨了Linux kernel Markers实现机制及其在kvmtrace的实际应用。

Kernel-based virtual machine事件跟踪机制的设计与实现

Network intrusion detection systems are faced with the challenge of identifying diverse attacks, in extremely high speed networks. For this reason, they must operate at multi-Gigabit speeds, while performing highly-complex per-packet and per-flow data processing. In this paper, we present a multi-parallel intrusion detection architecture tailored for high speed networks. To cope with the increased processing throughput requirements, our system parallelizes network traffic processing and analysis at three levels, using multi-queue NICs, multiple CPUs, and multiple GPUs. The proposed design avoids locking, optimizes data transfers between the different processing units, and speeds up data processing by mapping different operations to the processing units where they are best suited. Our experimental evaluation shows that our prototype implementation based on commodity off-the-shelf equipment can reach processing speeds of up to 5.2 Gbit/s with zero packet loss when analyzing traffic in a real network, whereas the pattern matching engine alone reaches speeds of up to 70 Gbit/s, which is an almost four times improvement over prior solutions that use specialized hardware.

/pdf/midea-a-multi-parallel-intrusion-detection-architecture-4kjokkvoc0.pdf

MIDeA: a multi-parallel intrusion detection architecture

Deep packet inspection (DPI) is widely used in content-aware network applications such as network intrusion detection systems, traffic billing, load balancing, and government surveillance. Pattern matching is a core and critical step in DPI, which checks the payload of each packet for known signatures (patterns) in order to identify packets with certain characteristics (e.g., malicious packets that carry viruses or worms). Regular expression is the major tool for signature description due to its powerful and flexible expressive ability. However, this flexibility also brings great challenges for efficient implementation in practice. Despite of hundreds to thousands of empirical proposals, wire-speed matching for large scale regular expressions still remains a big challenge. The gap between the matching throughput and the link speed is widening with the ever-increasing network link speed and pattern scale. This survey begins with a full-scale application background of DPI and technical background of regular expression matching in order to provide a global view and essential knowledge for readers. We then analyze the challenges in regular expression matching originated from the state explosion of finite state automaton used for regular expression matching. The nature of state explosion is analyzed in details, and the state-of-the-art solutions are grouped into categories of methods to relieve state expansion and methods to avoid state explosion, suggestions are also provided for building compact and efficient automata in different scenarios. Furthermore, proposals employing parallel platforms, including field-programmable gate array, GPU, general multi-processors, and ternary content addressable memory, to accelerate the matching process are introduced and thoroughly discussed. We also provide guidelines for efficient deployment for each of these platforms.

A Survey on Regular Expression Matching for Deep Packet Inspection: Applications, Algorithms, and Hardware Platforms

Network intrusion detection systems (NIDSs) are one of the latest developments in security. The matching of packet strings against collected signatures dominates signature-based NIDS performance. Network processors are also one of the fastest growing segments of the semiconductor market, because they are designed to provide scalable and flexible solutions that can accommodate change quickly and economically. This work presents a fast string-matching algorithm (called FNP) over the network processor platform that conducts matching sets of patterns in parallel. This design also supports numerous practical features such as case-sensitive string matching, signature prioritization, and multiple-content signatures. This efficient multiple-pattern matching algorithm utilizes the hardware facilities provided by typical network processors instead of employing the external lookup co-processors. To verify the efficiency and practicability of the proposed algorithm, it was implemented on the Vitesse IQ2000 network processor platform. The searching patterns used in the present experiments are derived from the well-known Snort ruleset cited by most open-source and commercial NIDSs. This work shows that combining our string-matching methodology, hashing engine supported by most network processors, and characteristics of current Snort signatures frequently improves performance and reduces number of memory accesses compared to conventional string-matching algorithms. Another contribution of this work is to highlight that, besides total number of searching patterns, shortest pattern length is also a major influence on NIDS multipattern matching algorithm performance.

A fast string-matching algorithm for network processor-based intrusion detection system

Network intrusion detection systems (NIDS) are one of the latest developments in security. The matching of packet strings against collected signatures dominates signature-based NIDS performance. This work presents FNP/sup 2/, an efficient pattern-matching engine designed for Network Processor platform which conducts matching sets of patterns in parallel. This work shows that combining our string matching methodology, hashing engine supported by most network processors, and characteristics of current Snort signatures frequently improves performance and reduces number of memory accesses compared to current NIDS pattern matching algorithms. Another contribution is to highlight that, besides total number of searching patterns, shortest pattern length is also a major influence on NIDS multi-pattern matching algorithm performance.

A fast pattern-match engine for network processor-based network intrusion detection system

Network intrusion detection systems (NIDS) monitor packets on the network and attempts to discover if a hacker is attempting to break into a system. The matching of packet strings against collected signatures dominates signature-based NIDS performance. Network processors are one of the fastest growing segments of the semiconductor market, because they are designed to provide scalable and flexible solutions that can accommodate change quickly and economically. This work presents a fast string matching algorithm (called FNP) over the network processor platform that conducts matching sets of patterns in parallel. FNP needs less number of memory accesses against conventional pattern-matching algorithms. Another contribution of this work is to highlight that, besides total number of searching patterns, shortest pattern length is also a major influence on NIDS multi-pattern matching algorithm performance.

A fast pattern matching algorithm for network processor-based intrusion detection system

In recent years, the size of virus signature databases has been growing rapidly, leading to a corresponding reduction in the performance of anti-virus (AV) software. In general, virus signature databases comprise string-based and hash-based (e.g., MD5) signatures. Currently the majority of signatures are hash-based and Cloud-based AV systems rely on them as the local cache to reduce the network loading. In this paper, we provide a novel scheme for looking up MD5 checksums to improve virus scanning performance involving hash-based signatures. The authors treat the range hash in which characters occur as a filter to avoid unnecessary lookups and keep the range of the exact search range to a minimum. The scheme is 135 times faster than ClamAV's in clean/general cases and only required 4MB of memory for hash-based filtering. This scheme could easily be extended to other hash-based applications.

A novel software-based MD5 checksum lookup scheme for anti-virus systems

Zero-day attack is a critical network attack. The zero-day attack period (ZDAP) is the period from the release of malware/exploit until a patch becomes available. IDS/IPS cannot effectively block zero-day attacks because they use pattern-based signatures in general. This paper proposes a Prophetic Defender (PD) by which ZDAP can be minimized. Prior to actual attack, hackers scan networks to identify hosts with vulnerable ports. If this port scanning can be detected early, zero-day attacks will become detectable. PD architecture makes use of a honeypot-based pseudo server deployed to detect malicious port scans. A port-scanning honeypot was operated by us in 6 years from 2009 to 2015. By analyzing the 6-year port-scanning log data, we understand that PD is effective for detecting and blocking zero-day attacks. The block rate of the proposed architecture is 98.5%.

Rong-Tai Liu

Papers

A fast string-matching algorithm for network processor-based intrusion detection system

A fast pattern-match engine for network processor-based network intrusion detection system

A fast pattern matching algorithm for network processor-based intrusion detection system

A novel software-based MD5 checksum lookup scheme for anti-virus systems

A predictive zero-day network defense using long-term port-scan recording