The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting
Neuroadaptive technologies are a type of passive Brain-computer interface (pBCI) that aim to incorporate implicit user-state information into human-machine interactions by monitoring neurophysiological signals. Evaluating machine learning and signal processing approaches represents a core aspect of...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-07-01
|
| Series: | Frontiers in Neuroergonomics |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fnrgo.2025.1582724/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849707482625081344 |
|---|---|
| author | Felix Schroeder Stephen Fairclough Frederic Dehais Matthew Richins |
| author_facet | Felix Schroeder Stephen Fairclough Frederic Dehais Matthew Richins |
| author_sort | Felix Schroeder |
| collection | DOAJ |
| description | Neuroadaptive technologies are a type of passive Brain-computer interface (pBCI) that aim to incorporate implicit user-state information into human-machine interactions by monitoring neurophysiological signals. Evaluating machine learning and signal processing approaches represents a core aspect of research into neuroadaptive technologies. These evaluations are often conducted under controlled laboratory settings and offline, where exhaustive analyses are possible. However, the manner in which classifiers are evaluated offline has been shown to impact reported accuracy levels, possibly biasing conclusions. In the current study, we investigated one of these sources of bias, the choice of cross-validation scheme, which is often not reported in sufficient detail. Across three independent electroencephalography (EEG) n-back datasets and 74 participants, we show how metrics and conclusions based on the same data can diverge with different cross-validation choices. A comparison of cross-validation schemes in which train and test subset boundaries either respect the block-structure of the data collection or not, illustrated how the relative performance of classifiers varies significantly with the evaluation method used. By computing bootstrapped 95% confidence intervals of differences across datasets, we showed that classification accuracies of Riemannian minimum distance (RMDM) classifiers may differ by up to 12.7% while those of a Filter Bank Common Spatial Pattern (FBCSP) based linear discriminant analysis (LDA) may differ by up to 30.4%. These differences across cross-validation implementations may impact the conclusions presented in research papers, which can complicate efforts to foster reproducibility. Our results exemplify why detailed reporting on data splitting procedures should become common practice. |
| format | Article |
| id | doaj-art-24dd47c7bb9a4c77ba63af0daa1aa99f |
| institution | DOAJ |
| issn | 2673-6195 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Neuroergonomics |
| spelling | doaj-art-24dd47c7bb9a4c77ba63af0daa1aa99f2025-08-20T03:15:54ZengFrontiers Media S.A.Frontiers in Neuroergonomics2673-61952025-07-01610.3389/fnrgo.2025.15827241582724The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reportingFelix Schroeder0Stephen Fairclough1Frederic Dehais2Matthew Richins3School of Psychology, Liverpool John Moores University, Liverpool, United KingdomSchool of Psychology, Liverpool John Moores University, Liverpool, United KingdomInstitut Supérieur de l'Aéronautique et de l'Espace (ISAE-SUPAERO), Université de Toulouse, Toulouse, FranceDefence Science and Technology Laboratory, Salisbury, United KingdomNeuroadaptive technologies are a type of passive Brain-computer interface (pBCI) that aim to incorporate implicit user-state information into human-machine interactions by monitoring neurophysiological signals. Evaluating machine learning and signal processing approaches represents a core aspect of research into neuroadaptive technologies. These evaluations are often conducted under controlled laboratory settings and offline, where exhaustive analyses are possible. However, the manner in which classifiers are evaluated offline has been shown to impact reported accuracy levels, possibly biasing conclusions. In the current study, we investigated one of these sources of bias, the choice of cross-validation scheme, which is often not reported in sufficient detail. Across three independent electroencephalography (EEG) n-back datasets and 74 participants, we show how metrics and conclusions based on the same data can diverge with different cross-validation choices. A comparison of cross-validation schemes in which train and test subset boundaries either respect the block-structure of the data collection or not, illustrated how the relative performance of classifiers varies significantly with the evaluation method used. By computing bootstrapped 95% confidence intervals of differences across datasets, we showed that classification accuracies of Riemannian minimum distance (RMDM) classifiers may differ by up to 12.7% while those of a Filter Bank Common Spatial Pattern (FBCSP) based linear discriminant analysis (LDA) may differ by up to 30.4%. These differences across cross-validation implementations may impact the conclusions presented in research papers, which can complicate efforts to foster reproducibility. Our results exemplify why detailed reporting on data splitting procedures should become common practice.https://www.frontiersin.org/articles/10.3389/fnrgo.2025.1582724/fullpassive Brain-Computer InterfacespBCIelectroencephalographyEEGcross-validationnon-stationarity |
| spellingShingle | Felix Schroeder Stephen Fairclough Frederic Dehais Matthew Richins The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting Frontiers in Neuroergonomics passive Brain-Computer Interfaces pBCI electroencephalography EEG cross-validation non-stationarity |
| title | The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting |
| title_full | The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting |
| title_fullStr | The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting |
| title_full_unstemmed | The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting |
| title_short | The impact of cross-validation choices on pBCI classification metrics: lessons for transparent reporting |
| title_sort | impact of cross validation choices on pbci classification metrics lessons for transparent reporting |
| topic | passive Brain-Computer Interfaces pBCI electroencephalography EEG cross-validation non-stationarity |
| url | https://www.frontiersin.org/articles/10.3389/fnrgo.2025.1582724/full |
| work_keys_str_mv | AT felixschroeder theimpactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT stephenfairclough theimpactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT fredericdehais theimpactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT matthewrichins theimpactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT felixschroeder impactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT stephenfairclough impactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT fredericdehais impactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting AT matthewrichins impactofcrossvalidationchoicesonpbciclassificationmetricslessonsfortransparentreporting |