Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics

Advanced Persistent Threats (APTs) involve attackers maintaining a long-term presence on victim systems, leading to the stealthy exfiltration of sensitive data during network transfers. Despite existing methods to detect and halt APT data exfiltration, these attacks continue to pose significant thre...

Full description

Saved in:
Bibliographic Details
Main Authors: Xiaojuan Cai, Haibo Zhang, Chuadhry Mujeeb Ahmed, Hiroshi Koide
Format: Article
Language:English
Published: IEEE 2025-01-01
Series:IEEE Access
Subjects:
Online Access:https://ieeexplore.ieee.org/document/10990168/
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1849322656489275392
author Xiaojuan Cai
Haibo Zhang
Chuadhry Mujeeb Ahmed
Hiroshi Koide
author_facet Xiaojuan Cai
Haibo Zhang
Chuadhry Mujeeb Ahmed
Hiroshi Koide
author_sort Xiaojuan Cai
collection DOAJ
description Advanced Persistent Threats (APTs) involve attackers maintaining a long-term presence on victim systems, leading to the stealthy exfiltration of sensitive data during network transfers. Despite existing methods to detect and halt APT data exfiltration, these attacks continue to pose significant threats to sensitive information and result in substantial commercial losses. Current approaches primarily focus on preemptive measures, which are insufficient once early-stage detection fails due to a lack of continuous monitoring. We propose an effective and efficient network monitoring method to address this gap and detect APT exfiltration during data transfer. Our approach assumes the presence of an undetected APT attacker within the victim system. We examine data exfiltration across three exfiltration traffic environments: exfiltration over command control channels, exfiltration over transfer size limitations, and their combinations. We introduce two detection metrics: Package Transfer Rate and Byte Transfer Rate. Utilizing these metrics, we measure network traffic, categorize APT attack environments, and train deep neural network models, named EDXGB, using ensembled decision trees to predict APT exfiltration. Our method is validated on two public datasets and compared against six baseline methods. Additionally, we simulate real-world exfiltration scenarios by creating three exfiltration traffic environments for each dataset. The results demonstrate that our method effectively detects APT exfiltration across various network environments, enhancing data protection and secure transfer. The code is open source and available at <uri>https://github.com/cxjuan/EDXGB-for-APT</uri>.
format Article
id doaj-art-fb71baacf6ee4516b77633ccddc55992
institution Kabale University
issn 2169-3536
language English
publishDate 2025-01-01
publisher IEEE
record_format Article
series IEEE Access
spelling doaj-art-fb71baacf6ee4516b77633ccddc559922025-08-20T03:49:17ZengIEEEIEEE Access2169-35362025-01-0113818038182210.1109/ACCESS.2025.356777210990168Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection MetricsXiaojuan Cai0https://orcid.org/0009-0009-4242-8420Haibo Zhang1https://orcid.org/0000-0002-4275-405XChuadhry Mujeeb Ahmed2https://orcid.org/0000-0003-3644-0465Hiroshi Koide3https://orcid.org/0009-0008-7111-8053Department of Information Science and Technology, Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka, JapanDepartment of Artificial Intelligence, Faculty of Computer Science and Systems Engineering, Kyushu Institute of Technology, Iizuka, Fukuoka, JapanSecure and Resilient Systems Group, School of Computing, Newcastle University, Newcastle upon Tyne, U.K.Section of Cyber Security for Information Systems, Research Institute for Information Technology, Kyushu University, Fukuoka, JapanAdvanced Persistent Threats (APTs) involve attackers maintaining a long-term presence on victim systems, leading to the stealthy exfiltration of sensitive data during network transfers. Despite existing methods to detect and halt APT data exfiltration, these attacks continue to pose significant threats to sensitive information and result in substantial commercial losses. Current approaches primarily focus on preemptive measures, which are insufficient once early-stage detection fails due to a lack of continuous monitoring. We propose an effective and efficient network monitoring method to address this gap and detect APT exfiltration during data transfer. Our approach assumes the presence of an undetected APT attacker within the victim system. We examine data exfiltration across three exfiltration traffic environments: exfiltration over command control channels, exfiltration over transfer size limitations, and their combinations. We introduce two detection metrics: Package Transfer Rate and Byte Transfer Rate. Utilizing these metrics, we measure network traffic, categorize APT attack environments, and train deep neural network models, named EDXGB, using ensembled decision trees to predict APT exfiltration. Our method is validated on two public datasets and compared against six baseline methods. Additionally, we simulate real-world exfiltration scenarios by creating three exfiltration traffic environments for each dataset. The results demonstrate that our method effectively detects APT exfiltration across various network environments, enhancing data protection and secure transfer. The code is open source and available at <uri>https://github.com/cxjuan/EDXGB-for-APT</uri>.https://ieeexplore.ieee.org/document/10990168/Advanced persistent threatdata exfiltrationdeep learningprivacy preserving
spellingShingle Xiaojuan Cai
Haibo Zhang
Chuadhry Mujeeb Ahmed
Hiroshi Koide
Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics
IEEE Access
Advanced persistent threat
data exfiltration
deep learning
privacy preserving
title Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics
title_full Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics
title_fullStr Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics
title_full_unstemmed Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics
title_short Detecting Advanced Persistent Threat Exfiltration With Ensemble Deep Learning Tree Models and Novel Detection Metrics
title_sort detecting advanced persistent threat exfiltration with ensemble deep learning tree models and novel detection metrics
topic Advanced persistent threat
data exfiltration
deep learning
privacy preserving
url https://ieeexplore.ieee.org/document/10990168/
work_keys_str_mv AT xiaojuancai detectingadvancedpersistentthreatexfiltrationwithensembledeeplearningtreemodelsandnoveldetectionmetrics
AT haibozhang detectingadvancedpersistentthreatexfiltrationwithensembledeeplearningtreemodelsandnoveldetectionmetrics
AT chuadhrymujeebahmed detectingadvancedpersistentthreatexfiltrationwithensembledeeplearningtreemodelsandnoveldetectionmetrics
AT hiroshikoide detectingadvancedpersistentthreatexfiltrationwithensembledeeplearningtreemodelsandnoveldetectionmetrics