A Framework for Evaluating Geomagnetic Indices Forecasting Models
Abstract The use of Deep Learning models to forecast geomagnetic storms is achieving great results. However, the evaluation of these models is mainly supported on generic regression metrics (such as the Root Mean Squared Error or the Coefficient of Determination), which are not able to properly capt...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2024-03-01
|
| Series: | Space Weather |
| Subjects: | |
| Online Access: | https://doi.org/10.1029/2024SW003868 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841536434203262976 |
|---|---|
| author | Armando Collado‐Villaverde Pablo Muñoz Consuelo Cid |
| author_facet | Armando Collado‐Villaverde Pablo Muñoz Consuelo Cid |
| author_sort | Armando Collado‐Villaverde |
| collection | DOAJ |
| description | Abstract The use of Deep Learning models to forecast geomagnetic storms is achieving great results. However, the evaluation of these models is mainly supported on generic regression metrics (such as the Root Mean Squared Error or the Coefficient of Determination), which are not able to properly capture the specific particularities of geomagnetic storms forecasting. Particularly, they do not provide insights during the high activity periods. To overcome this issue, we introduce the Binned Forecasting Error to provide a more accurate assessment of models' performance across the different intensity levels of a geomagnetic storm. This metric facilitates a robust comparison of different forecasting models, presenting a true representation of a model's predictive capabilities while being resilient to different storms duration. In this direction, for enabling fair comparison among models, it is important to standardize the sets of geomagnetic storms for model training, validation and testing. To do this, we have started from the current sets used in the literature for forecasting the SYM‐H, enriching them with newer storms not considered previously, focusing not only on disturbances caused by Coronal Mass Ejections but also addressing High‐Speed Streams. To operationalize the evaluation framework, a comparative study is conducted between a baseline neural network model and a persistence model, showcasing the effectiveness of the new metric in evaluating forecasting performance during intense geomagnetic storms. Finally, we propose the use of preliminary measurements from ACE to evaluate the model performance in settings closer to an operational real‐time scenario, where the forecasting models are expected to operate. |
| format | Article |
| id | doaj-art-bc9466f79bdf4b8e94417afdc2c358f9 |
| institution | Kabale University |
| issn | 1542-7390 |
| language | English |
| publishDate | 2024-03-01 |
| publisher | Wiley |
| record_format | Article |
| series | Space Weather |
| spelling | doaj-art-bc9466f79bdf4b8e94417afdc2c358f92025-01-14T16:30:31ZengWileySpace Weather1542-73902024-03-01223n/an/a10.1029/2024SW003868A Framework for Evaluating Geomagnetic Indices Forecasting ModelsArmando Collado‐Villaverde0Pablo Muñoz1Consuelo Cid2Department of Computer Engineering Universidad de Alcalá Madrid SpainDepartment of Computer Engineering Universidad de Alcalá Madrid SpainDepartment of Physics and Mathematics Universidad de Alcalá Madrid SpainAbstract The use of Deep Learning models to forecast geomagnetic storms is achieving great results. However, the evaluation of these models is mainly supported on generic regression metrics (such as the Root Mean Squared Error or the Coefficient of Determination), which are not able to properly capture the specific particularities of geomagnetic storms forecasting. Particularly, they do not provide insights during the high activity periods. To overcome this issue, we introduce the Binned Forecasting Error to provide a more accurate assessment of models' performance across the different intensity levels of a geomagnetic storm. This metric facilitates a robust comparison of different forecasting models, presenting a true representation of a model's predictive capabilities while being resilient to different storms duration. In this direction, for enabling fair comparison among models, it is important to standardize the sets of geomagnetic storms for model training, validation and testing. To do this, we have started from the current sets used in the literature for forecasting the SYM‐H, enriching them with newer storms not considered previously, focusing not only on disturbances caused by Coronal Mass Ejections but also addressing High‐Speed Streams. To operationalize the evaluation framework, a comparative study is conducted between a baseline neural network model and a persistence model, showcasing the effectiveness of the new metric in evaluating forecasting performance during intense geomagnetic storms. Finally, we propose the use of preliminary measurements from ACE to evaluate the model performance in settings closer to an operational real‐time scenario, where the forecasting models are expected to operate.https://doi.org/10.1029/2024SW003868machine learninggeomagnetic indices forecastingforecasting metricsevaluation frameworkoperational evaluation |
| spellingShingle | Armando Collado‐Villaverde Pablo Muñoz Consuelo Cid A Framework for Evaluating Geomagnetic Indices Forecasting Models Space Weather machine learning geomagnetic indices forecasting forecasting metrics evaluation framework operational evaluation |
| title | A Framework for Evaluating Geomagnetic Indices Forecasting Models |
| title_full | A Framework for Evaluating Geomagnetic Indices Forecasting Models |
| title_fullStr | A Framework for Evaluating Geomagnetic Indices Forecasting Models |
| title_full_unstemmed | A Framework for Evaluating Geomagnetic Indices Forecasting Models |
| title_short | A Framework for Evaluating Geomagnetic Indices Forecasting Models |
| title_sort | framework for evaluating geomagnetic indices forecasting models |
| topic | machine learning geomagnetic indices forecasting forecasting metrics evaluation framework operational evaluation |
| url | https://doi.org/10.1029/2024SW003868 |
| work_keys_str_mv | AT armandocolladovillaverde aframeworkforevaluatinggeomagneticindicesforecastingmodels AT pablomunoz aframeworkforevaluatinggeomagneticindicesforecastingmodels AT consuelocid aframeworkforevaluatinggeomagneticindicesforecastingmodels AT armandocolladovillaverde frameworkforevaluatinggeomagneticindicesforecastingmodels AT pablomunoz frameworkforevaluatinggeomagneticindicesforecastingmodels AT consuelocid frameworkforevaluatinggeomagneticindicesforecastingmodels |