Showing papers by "Richard P. Lippmann published in 1999"

Results of the DARPA 1998 offline intrusion detection evaluation

Abstract: DARPA sponsored the first realistic and systematic evaluation of research intrusion detection systems in 1998. As part of this evaluation, MIT Lincoln Laboratory developed a test network which simulated a medium-size government site. Background traffic was generated over two months using custom traffic generators which looked like 100’s of users on 1000’s of hosts performing a wide variety of tasks and generating a rich mixture of network traffic. While this background traffic was being generated, automated attacks were launched against three UNIX victim machines (SunOS, Solaris, Linux) located on the inside of this simulated government site behind a router. More than 300 instances of 38 different attacks were embedded in roughly two months of training data and two weeks of test data. Six DARPA research sites participated in a blind evaluation where test data was provided without specifying the location of embedded attacks. Results were analyzed by generating receiver operating characteristic curves (ROCs) to determine the attack detection rate as a function of the false alarm rate. Performance was evaluated for old attacks included in the training data, new attacks which only occurred in the test data, and novel new never-before-seen attacks developed specifically for this evaluation. Detection performance for the best systems was reasonable (above 60% correct) at a false alarm rate of 10 false alarms per day for both old and new probe attacks and attacks where a local user illegally becomes root (u2r). Intrusion detection systems trained on old probe or u2r attacks generalized well to other attacks in these same categories. Detection rates were worse, especially for new and novel denial of service (dos) attacks and attacks where a remote user illegally accesses a local host (r2l). Although detection accuracy for old attacks in these two categories was roughly 80%, detection accuracy for new and novel attacks was below 25% even at high false alarm rates. An intrusion detection system formed from the best components of the submitted systems performed much better than a baseline keyword spotting system that is similar to many commercial and government systems. Across all 120 attacks in the test data, it reduced the false alarm rate by more than two orders of magnitude (from roughly 600 false alarms per day to 6) and it also increased the detection accuracy (from roughly 20% detections to 60%). These results suggest that future intrusion detection research should move towards developing algorithms that find new attacks and away from older approaches that focus on creating rules to find attack signatures. The current 1999 DARPA evaluation is extending the 1998 evaluation by adding Windows/NT victims, including new Windows/NT attacks, including insider attacks,

Evaluating Intrusion Detection Systems Without Attacking Your Friends: The 1998 DARPA Intrusion Detection Evaluation

Abstract: : Intrusion detection systems monitor the use of computers and the network over which they communicate, searching for unauthorized use, anomalous behavior, and attempts to deny users, machines or portions of the network access to services. Potential users of such systems need information that is rarely found in marketing literature, including how well a given system finds intruders and how much work is required to use and maintain that system in a fully functioning network with significant daily traffic. Researchers and developers can specify which prototypical attacks can be found by their systems, but without access to the normal traffic generated by day-to-day work, they can not describe how well their systems detect real attacks while passing background traffic and avoiding false alarms. This information is critical: every declared intrusion requires time to review, regardless of whether it is a correct detection for which a real intrusion occurred, or whether it is merely a false alarm. To meet the needs of researchers, developers and ultimately system administrators we have developed the first objective, repeatable, and realistic measurement of intrusion detection system performance. Network traffic on an Air Force base was measured, characterized and subsequently simulated on an isolated network on which a few computers were used to simulate thousands of different Unix systems and hundreds of different users during periods of normal network traffic. Simulated attackers mapped the network, issued denial of service attacks, illegally gained access to systems, and obtained super-user privileges. Attack types ranged from old, well-known attacks, to new, stealthy attacks. Seven weeks of training data and two weeks of testing data were generated, filling more than 30 CD-ROMs. Methods and results from the 1998 DARPA intrusion detection evaluation will be highlighted, and preliminary plans for the 1999 evaluation will be presented.