Deep Learning Models for Detection and Classification of Polymorphic Malware Over Encrypted Networks consisting of 2D CNN-LSTM, GAN-based adversarial training and GRAD-CAM for polymorphic malware detection explainability

DOI

10.2991/978-94-6463-970-4_19How to use a DOI?

Keywords

Polymorphic malware; encrypted traffic detection; CNN-LSTM; GAN adversarial training; GRAD-CAM explainability; deep learning cybersecurity

Abstract

Deep learning has emerged as a powerful tool for detecting polymorphic malware in encrypted network traffic, where traditional signature-based methods often fail due to encryption and code obfuscation. This study introduces a hybrid 2D CNN-LSTM model augmented with GAN-based adversarial training and GRAD-CAM explainability for resilient and comprehensible malware classification. The proposed method has an accuracy of 99.8%, a precision of 0.996%, and a recall of 99.0%. This shows that it can almost perfectly tell the difference between benign (1) and malicious (0) samples while keeping the false positive rate low (0.4%). The CNN part gets spatial features from encrypted payloads (such byte entropy and TLS handshake anomalies), and the LSTM layer gets temporal behavioral patterns. Adversarial training using GAN-generated malware variants greatly enhances generalization against polymorphic evasion strategies. Furthermore, GRAD-CAM visualizations provide critical explainability by highlighting malicious regions in encrypted traffic, enabling security analysts to validate detection logic. With a detection latency of just 2.86 ms, the model is suitable for real-time deployment in high-speed networks. Comparative investigation demonstrates superiority over conventional antivirus solutions (85–92% accuracy) and machine learning-based detectors (93–97% accuracy), especially in the management of encrypted threats. This study fills the gap between performance and transparency in malware detection by providing a scalable, efficient, and understandable answer for today’s cybersecurity problems.

Copyright

© 2025 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Ogba Paul
AU  - Timothy Moses
PY  - 2025
DA  - 2025/12/31
TI  - Deep Learning Models for Detection and Classification of Polymorphic Malware Over Encrypted Networks consisting of 2D CNN-LSTM, GAN-based adversarial training and GRAD-CAM for polymorphic malware detection explainability
BT  - Proceedings of the International Conference on Engineering, Science, and Urban Sustainability (ICESUS 2025)
PB  - Atlantis Press
SP  - 307
EP  - 321
SN  - 2352-5401
UR  - https://doi.org/10.2991/978-94-6463-970-4_19
DO  - 10.2991/978-94-6463-970-4_19
ID  - Paul2025
ER  -