Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions
Tropical cyclones (TCs) are destructive meteorological phenomena, necessitating accurate predictions of TC track and intensity to reduce risks to human life. This study evaluates three Transformer-based models – vanilla Transformer (Transformer), inverted Transformer (iTransformer), and temporal-var...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2025-12-01
|
| Series: | Engineering Applications of Computational Fluid Mechanics |
| Subjects: | |
| Online Access: | https://www.tandfonline.com/doi/10.1080/19942060.2025.2538180 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849705937986650112 |
|---|---|
| author | Yuan-Jiang Zeng Yi-Qing Ni Zheng-Wei Chen Guang-Zhi Zeng Jia-Yao Wang Pak-Wai Chan |
| author_facet | Yuan-Jiang Zeng Yi-Qing Ni Zheng-Wei Chen Guang-Zhi Zeng Jia-Yao Wang Pak-Wai Chan |
| author_sort | Yuan-Jiang Zeng |
| collection | DOAJ |
| description | Tropical cyclones (TCs) are destructive meteorological phenomena, necessitating accurate predictions of TC track and intensity to reduce risks to human life. This study evaluates three Transformer-based models – vanilla Transformer (Transformer), inverted Transformer (iTransformer), and temporal-variate Transformer (TVFormer) – which are trained, validated, and tested on best track data from 1980 to 2021 from the China Meteorological Administration for TC prediction, integrating temporal, variate, and hybrid token embeddings to analyze temporal and variate correlations. Comparative analysis with four recurrent neural network (RNN) models demonstrates the superiority of the refined Transformer models over RNNs: iTransformer reduces mean absolute error (MAE) and root mean square error (RMSE) by 29.55% and 25.80% (latitude), 50.31% and 46.18% (longitude), 8.71% and 9.98% (pressure), and 8.68% and 9.45% (wind speed), while TVFormer achieves MAE and RMSE reductions of 13.98% and 13.84% (latitude), 39.11% and 38.02% (longitude), 13.69% and 14.02% (pressure), and 12.84% and 12.94% (wind speed) on average. Among Transformer variants, iTransformer excels in track prediction, outperforming Transformer with 21.74% lower MAE and 18.26% lower RMSE for latitude, and 32.73% lower MAE and 24.01% lower RMSE for longitude. TVFormer dominates intensity prediction, reducing pressure errors by 4.42% (MAE) and 3.92% (RMSE) and wind speed errors by 19.21% (MAE) and 14.79% (RMSE) compared to Transformer, while outperforming iTransformer with 4.59% lower MAE and 3.68% lower RMSE for pressure and 3.83% lower MAE and 3.18% lower RMSE for wind speed. Notably, TVFormer also enhances track prediction, with 7.10% reduction in MAE and 7.02% reduction in RMSE for latitude, and 22.84% reduction in MAE and 17.09% reduction in RMSE for longitude compared to Transformer. These results highlight the superiority of iTransformer in track prediction and the efficacy of TVFormer in intensity prediction, thanks to their ability to exploit temporal and variate dependencies, offering potential for TC disaster preparedness systems. |
| format | Article |
| id | doaj-art-e29f65e886ea4a71af43a1d20eba9976 |
| institution | DOAJ |
| issn | 1994-2060 1997-003X |
| language | English |
| publishDate | 2025-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Engineering Applications of Computational Fluid Mechanics |
| spelling | doaj-art-e29f65e886ea4a71af43a1d20eba99762025-08-20T03:16:21ZengTaylor & Francis GroupEngineering Applications of Computational Fluid Mechanics1994-20601997-003X2025-12-0119110.1080/19942060.2025.2538180Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictionsYuan-Jiang Zeng0Yi-Qing Ni1Zheng-Wei Chen2Guang-Zhi Zeng3Jia-Yao Wang4Pak-Wai Chan5Department of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong, People’s Republic of ChinaDepartment of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong, People’s Republic of ChinaDepartment of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong, People’s Republic of ChinaDepartment of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong, People’s Republic of ChinaDepartment of Civil and Environmental Engineering, The Hong Kong Polytechnic University, Hong Kong, People’s Republic of ChinaHong Kong Observatory, Hong Kong, People’s Republic of ChinaTropical cyclones (TCs) are destructive meteorological phenomena, necessitating accurate predictions of TC track and intensity to reduce risks to human life. This study evaluates three Transformer-based models – vanilla Transformer (Transformer), inverted Transformer (iTransformer), and temporal-variate Transformer (TVFormer) – which are trained, validated, and tested on best track data from 1980 to 2021 from the China Meteorological Administration for TC prediction, integrating temporal, variate, and hybrid token embeddings to analyze temporal and variate correlations. Comparative analysis with four recurrent neural network (RNN) models demonstrates the superiority of the refined Transformer models over RNNs: iTransformer reduces mean absolute error (MAE) and root mean square error (RMSE) by 29.55% and 25.80% (latitude), 50.31% and 46.18% (longitude), 8.71% and 9.98% (pressure), and 8.68% and 9.45% (wind speed), while TVFormer achieves MAE and RMSE reductions of 13.98% and 13.84% (latitude), 39.11% and 38.02% (longitude), 13.69% and 14.02% (pressure), and 12.84% and 12.94% (wind speed) on average. Among Transformer variants, iTransformer excels in track prediction, outperforming Transformer with 21.74% lower MAE and 18.26% lower RMSE for latitude, and 32.73% lower MAE and 24.01% lower RMSE for longitude. TVFormer dominates intensity prediction, reducing pressure errors by 4.42% (MAE) and 3.92% (RMSE) and wind speed errors by 19.21% (MAE) and 14.79% (RMSE) compared to Transformer, while outperforming iTransformer with 4.59% lower MAE and 3.68% lower RMSE for pressure and 3.83% lower MAE and 3.18% lower RMSE for wind speed. Notably, TVFormer also enhances track prediction, with 7.10% reduction in MAE and 7.02% reduction in RMSE for latitude, and 22.84% reduction in MAE and 17.09% reduction in RMSE for longitude compared to Transformer. These results highlight the superiority of iTransformer in track prediction and the efficacy of TVFormer in intensity prediction, thanks to their ability to exploit temporal and variate dependencies, offering potential for TC disaster preparedness systems.https://www.tandfonline.com/doi/10.1080/19942060.2025.2538180Tropical cyclonestrackintensityTransformertoken embeddings |
| spellingShingle | Yuan-Jiang Zeng Yi-Qing Ni Zheng-Wei Chen Guang-Zhi Zeng Jia-Yao Wang Pak-Wai Chan Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions Engineering Applications of Computational Fluid Mechanics Tropical cyclones track intensity Transformer token embeddings |
| title | Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions |
| title_full | Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions |
| title_fullStr | Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions |
| title_full_unstemmed | Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions |
| title_short | Investigation of the impact of token embeddings in Transformer-based models on short-term tropical cyclone track and intensity predictions |
| title_sort | investigation of the impact of token embeddings in transformer based models on short term tropical cyclone track and intensity predictions |
| topic | Tropical cyclones track intensity Transformer token embeddings |
| url | https://www.tandfonline.com/doi/10.1080/19942060.2025.2538180 |
| work_keys_str_mv | AT yuanjiangzeng investigationoftheimpactoftokenembeddingsintransformerbasedmodelsonshorttermtropicalcyclonetrackandintensitypredictions AT yiqingni investigationoftheimpactoftokenembeddingsintransformerbasedmodelsonshorttermtropicalcyclonetrackandintensitypredictions AT zhengweichen investigationoftheimpactoftokenembeddingsintransformerbasedmodelsonshorttermtropicalcyclonetrackandintensitypredictions AT guangzhizeng investigationoftheimpactoftokenembeddingsintransformerbasedmodelsonshorttermtropicalcyclonetrackandintensitypredictions AT jiayaowang investigationoftheimpactoftokenembeddingsintransformerbasedmodelsonshorttermtropicalcyclonetrackandintensitypredictions AT pakwaichan investigationoftheimpactoftokenembeddingsintransformerbasedmodelsonshorttermtropicalcyclonetrackandintensitypredictions |