Due to temporary inabilities of the models to match the real values with the predictions, random spikes can arise in the “alarm” time series. Thus, in order to make the alarm system more reliable, we use a two-level structure: this first alarm, the one defined above, is merely a warning signal and is processed again to produce a more accurate second level alarm signal. Using a Moving Aggregation node, the moving averages are calculated on a backward window of 21 samples of the level 1 alarm (warning) signals. This moving average operation smooths out all short random spikes in the level 1 alarm time series, retaining only the ones that persist over time.
is paper presents DeepLog, a general-purpose framework for
online log anomaly detection and diagnosis using a deep neural
network based approach. DeepLog learns and encodes entire log
message including timestamp, log key, and parameter values. It
performs anomaly detection at per log entry level, rather than at
per session level as many previous methods are limited to. DeepLog
can separate out dierent tasks from a log le and construct a work-
ow model for each task using both deep learning (LSTM) and
classic mining (density clustering) approaches. is enables eective
anomaly diagnosis. By incorporating user feedback, DeepLog
supports online update/training to its LSTM models, hence is able
to incorporate and adapt to new execution paerns. Extensive evaluation
on large system logs have clearly demonstrated the superior
eectiveness of DeepLog compared with previous methods.
Future work include but are not limited to incorporating other
types of RNNs (recurrent neural networks) into DeepLog to test
their eciency, and integrating log data from dierent applications
and systems to perform more comprehensive system diagnosis (e.g.,
failure of a MySQL database may be caused by a disk failure as
reected in a separate system log).
online log anomaly detection and diagnosis using a deep neural
network based approach. DeepLog learns and encodes entire log
message including timestamp, log key, and parameter values. It
performs anomaly detection at per log entry level, rather than at
per session level as many previous methods are limited to. DeepLog
can separate out dierent tasks from a log le and construct a work-
ow model for each task using both deep learning (LSTM) and
classic mining (density clustering) approaches. is enables eective
anomaly diagnosis. By incorporating user feedback, DeepLog
supports online update/training to its LSTM models, hence is able
to incorporate and adapt to new execution paerns. Extensive evaluation
on large system logs have clearly demonstrated the superior
eectiveness of DeepLog compared with previous methods.
Future work include but are not limited to incorporating other
types of RNNs (recurrent neural networks) into DeepLog to test
their eciency, and integrating log data from dierent applications
and systems to perform more comprehensive system diagnosis (e.g.,
failure of a MySQL database may be caused by a disk failure as
reected in a separate system log).
