Abstract
Streaming anomaly detection is a difficult task because the data distribution is changing, and concept drift degrades the performance of traditional detection methods. To solve the issue, this article suggests the HAD-CDA (Hybrid Anomaly Detection with Concept Drift Adaptation), which is a combined system that can detect local anomalies in the time series with LSTM Autoencoders (LSTM-AE) and global anomalies in the distribution with Quant Tree-EWMA (QT-EWMA). The proposed framework accomplishes three objectives: (i) the use of the dynamic weighting mechanism, which automatically changes the contribution of each component (lambda between 0.2 and 0.8) according to their effectiveness, (ii) two concept drift detectors are proposed, i.e., the Kolmogorov-Smirnov and Page-Hinkley tests, to allow detecting concept drift; and (iii) the use of Elastic Weight Consolidation (EWC) to reduce catastrophic forgetting during update of the model. Experiments on four real-world streaming datasets, HTTP, SMTP, ForestCover, and Shuttle, indicate that HAD-CDA has AUCs of 0.95-0.97, an 8-9% improvement over state-of-the-art methods, and F1-scores of 0.81-0.95. The recall measure obtained by the LSTM-AE element is 0.8094, compared to 0.9399, and the specificity of QT-EWMA is very high at 0.9399. The framework is highly adaptable to different types of drifts, regaining around 9092 performance levels before drift, and 1525 windows would be needed by a baseline method, with a low processing latency of 12.430.9 ms per window. Having a per-sequence complexity of O(1), memory cost of O(n), low DIS (approximately 0.08), and stability indices of 0.02-0.04, the suggested HAD-CDA framework is an accurate, efficient, and robust solution to real-world streaming anomaly detection in changing data