Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum
Automatic sleep staging is crucial for diagnosing sleep disorders, however, existing inter-epoch feature extraction schemes such as RNN-based networks or transformers often struggle with long sleep sequences due to overfitting. This study presents a novel automatic sleep staging method utilizing a p...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10966867/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849714432494534656 |
|---|---|
| author | Hyounggyu Kim Moogyeong Kim Wonzoo Chung |
| author_facet | Hyounggyu Kim Moogyeong Kim Wonzoo Chung |
| author_sort | Hyounggyu Kim |
| collection | DOAJ |
| description | Automatic sleep staging is crucial for diagnosing sleep disorders, however, existing inter-epoch feature extraction schemes such as RNN-based networks or transformers often struggle with long sleep sequences due to overfitting. This study presents a novel automatic sleep staging method utilizing a pre-trained vision transformer with compression as a sequence encoder and a two-step attention to enhance the sleep-stage classification performance. In contrast to existing transformer-based methods, the pre-trained transformer with compression can handle long sequences covering a sleep cycle, leveraging robust feature extraction capabilities with substantially fewer parameters. Furthermore, an epoch encoder based on a bidirectional temporal convolutional network with a multi-head two-step attention mechanism is proposed to improve the efficiency of epoch-level feature extraction. The performance of the proposed method is evaluated using three publicly available datasets: SleepEDF-20, SleepEDF-78, and SHHS. Numerical experiments show notable performance enhancement of the proposed scheme in comparison with the state-of-the-art algorithms, particularly for small training datasets, which validates the resilience of the proposed method against overfitting. These results suggest that with appropriate regularization, transformer-based models can effectively capture long-term contextual information across a complete sleep cycle. |
| format | Article |
| id | doaj-art-c24c6153bb9541dbaeb2892cb012d836 |
| institution | DOAJ |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-c24c6153bb9541dbaeb2892cb012d8362025-08-20T03:13:42ZengIEEEIEEE Access2169-35362025-01-0113696506965910.1109/ACCESS.2025.356131010966867Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted SumHyounggyu Kim0https://orcid.org/0009-0003-1628-2073Moogyeong Kim1https://orcid.org/0000-0002-7457-9639Wonzoo Chung2https://orcid.org/0000-0001-7381-250XDepartment of Artificial Intelligence, Korea University, Seoul, South KoreaDepartment of Artificial Intelligence, Korea University, Seoul, South KoreaDepartment of Artificial Intelligence, Korea University, Seoul, South KoreaAutomatic sleep staging is crucial for diagnosing sleep disorders, however, existing inter-epoch feature extraction schemes such as RNN-based networks or transformers often struggle with long sleep sequences due to overfitting. This study presents a novel automatic sleep staging method utilizing a pre-trained vision transformer with compression as a sequence encoder and a two-step attention to enhance the sleep-stage classification performance. In contrast to existing transformer-based methods, the pre-trained transformer with compression can handle long sequences covering a sleep cycle, leveraging robust feature extraction capabilities with substantially fewer parameters. Furthermore, an epoch encoder based on a bidirectional temporal convolutional network with a multi-head two-step attention mechanism is proposed to improve the efficiency of epoch-level feature extraction. The performance of the proposed method is evaluated using three publicly available datasets: SleepEDF-20, SleepEDF-78, and SHHS. Numerical experiments show notable performance enhancement of the proposed scheme in comparison with the state-of-the-art algorithms, particularly for small training datasets, which validates the resilience of the proposed method against overfitting. These results suggest that with appropriate regularization, transformer-based models can effectively capture long-term contextual information across a complete sleep cycle.https://ieeexplore.ieee.org/document/10966867/Automatic sleep stagingelectroencephalogramlong-term dependency modelingsequence-to-sequencevision transformertemporal convolutional network |
| spellingShingle | Hyounggyu Kim Moogyeong Kim Wonzoo Chung Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum IEEE Access Automatic sleep staging electroencephalogram long-term dependency modeling sequence-to-sequence vision transformer temporal convolutional network |
| title | Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum |
| title_full | Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum |
| title_fullStr | Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum |
| title_full_unstemmed | Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum |
| title_short | Sleep Staging Using Compressed Vision Transformer With Novel Two-Step Attention Weighted Sum |
| title_sort | sleep staging using compressed vision transformer with novel two step attention weighted sum |
| topic | Automatic sleep staging electroencephalogram long-term dependency modeling sequence-to-sequence vision transformer temporal convolutional network |
| url | https://ieeexplore.ieee.org/document/10966867/ |
| work_keys_str_mv | AT hyounggyukim sleepstagingusingcompressedvisiontransformerwithnoveltwostepattentionweightedsum AT moogyeongkim sleepstagingusingcompressedvisiontransformerwithnoveltwostepattentionweightedsum AT wonzoochung sleepstagingusingcompressedvisiontransformerwithnoveltwostepattentionweightedsum |