Peek Inside the Closed World: Evaluating Autoencoder-Based Detection of DDoS to Cloud
Machine-learning-based anomaly detection (ML-based AD) has been successful at detecting DDoS events in the lab. However published evaluations of ML-based AD have used only limited data and provided minimal insight into why it works. To address limited evaluation against real-world data, we apply autoencoder, an existing ML-AD model, to 57 DDoS attack events captured at 5 cloud IPs from a major cloud provider. We show that our models detect nearly all malicious flows for 2 of the 4 cloud IPs under attack (at least 99.99%) and detect most malicious flows (94.75% and 91.37%) for the remaining 2 IPs. Our models also maintain near-zero false positives on benign flows to all 5 IPs. Our primary contribution is to improve our understanding for why ML-based AD works on some malicious flows but not others. We interpret our detection results with feature attribution and counterfactual explanation. We show that our models are better at detecting malicious flows with anomalies on allow-listed features (those with only a few benign values) than flows with anomalies on deny-listed features (those with mostly benign values) because our models are more likely to learn correct normality for allow-listed features. We then show that our models are better at detecting malicious flows with anomalies on unordered features (that have no ordering among their values) than flows with anomalies on ordered features because even with incomplete normality, our models could still detect anomalies on unordered feature with high recall. Lastly, we summarize the implications of what we learn on applying autoencoder-based AD in production: training with noisy real-world data is possible, autoencoder can reliably detect real-world anomalies on well-represented unordered features and combinations of autoencoder-based AD and heuristic-based filters can help both.