Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection
The widespread adoption of encryption in computer networks has made detecting malicious traffic, especially at network perimeters, increasingly challenging. As packet contents are concealed, traditional monitoring techniques such as Deep Packet Inspection (DPI) become ineffective. Consequently, rese...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11113262/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849405825917911040 |
|---|---|
| author | A. Ramzi Bahlali Abdelmalik Bachir Abdeldjalil Labed |
| author_facet | A. Ramzi Bahlali Abdelmalik Bachir Abdeldjalil Labed |
| author_sort | A. Ramzi Bahlali |
| collection | DOAJ |
| description | The widespread adoption of encryption in computer networks has made detecting malicious traffic, especially at network perimeters, increasingly challenging. As packet contents are concealed, traditional monitoring techniques such as Deep Packet Inspection (DPI) become ineffective. Consequently, researchers have started employing data-driven methods based on Machine and Deep Learning (ML & DL) to identify malicious behavior even from encrypted traffic, typically within Anomaly-based Network Intrusion Detection Systems (A-NIDS). Existing approaches rely heavily on supervised learning, which requires large volumes of labeled benign and malicious traffic. Generating these labels is time-consuming, error-prone, and often requires expert knowledge. In this paper, we propose a semi-supervised learning framework that leverages Self-Supervised Learning (SSL) to learn discriminative representations from unlabeled network traffic. We design a novel pretext task that predicts important masked features, enabling the model to capture meaningful structure in the data. The learned representations are fine-tuned with minimal labeled data using a Custom-Autoencoder (Custom-AE) classifier. Experimental results show that the representation learned from our proposed pretext task outperforms the best competing method in terms of accuracy by 3.41% on UNSW-NB15 (NB15) and 1.53% on CSE-CIC-IDS2018 (CSE18) when evaluated using linear probing. When fine-tuned with the Custom-AE on only 100 benign and 10 malicious samples, it achieves 83.51% (NB15) and 87.43% (CSE18) accuracy, representing gains of 4.55% and 5.08% over the initial features, respectively. This demonstrates stronger suitability for label-scarce real-world scenarios compared to existing approaches. |
| format | Article |
| id | doaj-art-859fa2b9eeba468ebcd0b41bd043c7ea |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-859fa2b9eeba468ebcd0b41bd043c7ea2025-08-20T03:36:34ZengIEEEIEEE Access2169-35362025-01-011313914113915410.1109/ACCESS.2025.359617911113262Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly DetectionA. Ramzi Bahlali0https://orcid.org/0000-0001-5189-8541Abdelmalik Bachir1https://orcid.org/0000-0001-5160-9412Abdeldjalil Labed2https://orcid.org/0009-0007-0348-3780IMATH Laboratory, University of Toulon, La Garde, FranceNational School of Artificial Intelligence, Algiers, AlgeriaMediterranean Institute of Technology, South Mediterranean University, Tunis, TunisiaThe widespread adoption of encryption in computer networks has made detecting malicious traffic, especially at network perimeters, increasingly challenging. As packet contents are concealed, traditional monitoring techniques such as Deep Packet Inspection (DPI) become ineffective. Consequently, researchers have started employing data-driven methods based on Machine and Deep Learning (ML & DL) to identify malicious behavior even from encrypted traffic, typically within Anomaly-based Network Intrusion Detection Systems (A-NIDS). Existing approaches rely heavily on supervised learning, which requires large volumes of labeled benign and malicious traffic. Generating these labels is time-consuming, error-prone, and often requires expert knowledge. In this paper, we propose a semi-supervised learning framework that leverages Self-Supervised Learning (SSL) to learn discriminative representations from unlabeled network traffic. We design a novel pretext task that predicts important masked features, enabling the model to capture meaningful structure in the data. The learned representations are fine-tuned with minimal labeled data using a Custom-Autoencoder (Custom-AE) classifier. Experimental results show that the representation learned from our proposed pretext task outperforms the best competing method in terms of accuracy by 3.41% on UNSW-NB15 (NB15) and 1.53% on CSE-CIC-IDS2018 (CSE18) when evaluated using linear probing. When fine-tuned with the Custom-AE on only 100 benign and 10 malicious samples, it achieves 83.51% (NB15) and 87.43% (CSE18) accuracy, representing gains of 4.55% and 5.08% over the initial features, respectively. This demonstrates stronger suitability for label-scarce real-world scenarios compared to existing approaches.https://ieeexplore.ieee.org/document/11113262/Autoencoder (AE)anomaly detection (AD)deep learning (DL)intrusion detectionself-supervised learning (SSL)semi-supervised Learning |
| spellingShingle | A. Ramzi Bahlali Abdelmalik Bachir Abdeldjalil Labed Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection IEEE Access Autoencoder (AE) anomaly detection (AD) deep learning (DL) intrusion detection self-supervised learning (SSL) semi-supervised Learning |
| title | Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection |
| title_full | Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection |
| title_fullStr | Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection |
| title_full_unstemmed | Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection |
| title_short | Self-Supervised Learning Meets Custom Autoencoder Classifier: A Semi-Supervised Approach for Encrypted Traffic Anomaly Detection |
| title_sort | self supervised learning meets custom autoencoder classifier a semi supervised approach for encrypted traffic anomaly detection |
| topic | Autoencoder (AE) anomaly detection (AD) deep learning (DL) intrusion detection self-supervised learning (SSL) semi-supervised Learning |
| url | https://ieeexplore.ieee.org/document/11113262/ |
| work_keys_str_mv | AT aramzibahlali selfsupervisedlearningmeetscustomautoencoderclassifierasemisupervisedapproachforencryptedtrafficanomalydetection AT abdelmalikbachir selfsupervisedlearningmeetscustomautoencoderclassifierasemisupervisedapproachforencryptedtrafficanomalydetection AT abdeldjalillabed selfsupervisedlearningmeetscustomautoencoderclassifierasemisupervisedapproachforencryptedtrafficanomalydetection |