Proceedings ArticleDOI
Clustering event logs using iterative partitioning
Adetokunbo Makanju,A. Nur Zincir-Heywood,Evangelos E. Milios +2 more
- pp 1255-1264
Reads0
Chats0
TLDR
This paper presents IPLoM (Iterative Partitioning Log Mining), a novel algorithm for the mining of clusters from event logs that outperforms the other algorithms statistically significantly, and is also able to achieve an average F- Measure performance 78% when the closest other algorithm achieves an F-Measure performance of 10%.Abstract:
The importance of event logs, as a source of information in systems and network management cannot be overemphasized. With the ever increasing size and complexity of today's event logs, the task of analyzing event logs has become cumbersome to carry out manually. For this reason recent research has focused on the automatic analysis of these log files. In this paper we present IPLoM (Iterative Partitioning Log Mining), a novel algorithm for the mining of clusters from event logs. Through a 3-Step hierarchical partitioning process IPLoM partitions log data into its respective clusters. In its 4th and final stage IPLoM produces cluster descriptions or line formats for each of the clusters produced. Unlike other similar algorithms IPLoM is not based on the Apriori algorithm and it is able to find clusters in data whether or not its instances appear frequently. Evaluations show that IPLoM outperforms the other algorithms statistically significantly, and it is also able to achieve an average F-Measure performance 78% when the closest other algorithm achieves an F-Measure performance of 10%.read more
Citations
More filters
Proceedings ArticleDOI
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
TL;DR: DeepLog, a deep neural network model utilizing Long Short-Term Memory (LSTM), is proposed, to model a system log as a natural language sequence, which allows DeepLog to automatically learn log patterns from normal execution, and detect anomalies when log patterns deviate from the model trained from log data under normal execution.
Proceedings ArticleDOI
Detecting large-scale system problems by mining console logs
TL;DR: In this article, a general methodology to mine this rich source of information to automatically detect system runtime problems was proposed, combining source code analysis with information retrieval to create composite features and then analyze these features using machine learning to detect operational problems.
Proceedings Article
Detecting Large-Scale System Problems by Mining Console Logs
TL;DR: This work first parse console logs by combining source code analysis with information retrieval to create composite features, and then analyzes these features using machine learning to detect operational problems to automatically detect system runtime problems.
Proceedings ArticleDOI
Experience Report: System Log Analysis for Anomaly Detection
TL;DR: A detailed review and evaluation of six state-of-the-art log-based anomaly detection methods, including three supervised methods and three unsupervised methods, and also releases an open-source toolkit allowing ease of reuse.
Proceedings ArticleDOI
Tools and benchmarks for automated log parsing
TL;DR: This paper presents a comprehensive evaluation study on automated log parsing, evaluating 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software and reports the results in terms of accuracy, robustness, and efficiency.
References
More filters
Journal ArticleDOI
Mining frequent patterns without candidate generation
Jiawei Han,Jian Pei,Yiwen Yin +2 more
TL;DR: This study proposes a novel frequent pattern tree (FP-tree) structure, which is an extended prefix-tree structure for storing compressed, crucial information about frequent patterns, and develops an efficient FP-tree-based mining method, FP-growth, for mining the complete set of frequent patterns by pattern fragment growth.
Proceedings ArticleDOI
A data clustering algorithm for mining patterns from event logs
TL;DR: A novel clustering algorithm for log file data sets is presented which helps one to detect frequent patterns from log files, to build log file profiles, and to identify anomalous log file lines.
Proceedings ArticleDOI
Mining partially periodic event patterns with unknown periods
Sheng Ma,Joseph L. Hellerstein +1 more
TL;DR: This work develops two algorithms for mining p-patterns based on the order in which the aforementioned sub-tasks are performed: the period-first algorithm and the association- first algorithm, and develops a novel approach based on a chi-squared test.
The BSD Syslog Protocol
TL;DR: This document describes the observed behavior of the syslog protocol, a protocol used for the transmission of event notification messages across networks for many years that has been ported to many other operating systems as well as being embedded into many other networked devices.
Proceedings ArticleDOI
Using Hidden Semi-Markov Models for Effective Online Failure Prediction
Felix Salfner,Miroslaw Malek +1 more
TL;DR: This work focuses on methods that use event-driven sources such as errors for online failure prediction and uses hidden semi-Markov models (HSMMs) for this purpose and demonstrates effectiveness based on field data of a commercial telecommunication system.