Benchmarking Methods for Pointwise Reliability
The growing interest in machine learning in a critical domain like healthcare emphasizes the need for reliable predictions, as decisions based on these outputs can have significant consequences. This study benchmarks methods for assessing pointwise reliability, focusing on data-driven techniques bas...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-04-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/16/4/327 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850181009481400320 |
|---|---|
| author | Cláudio Correia Simão Paredes Teresa Rocha Jorge Henriques Jorge Bernardino |
| author_facet | Cláudio Correia Simão Paredes Teresa Rocha Jorge Henriques Jorge Bernardino |
| author_sort | Cláudio Correia |
| collection | DOAJ |
| description | The growing interest in machine learning in a critical domain like healthcare emphasizes the need for reliable predictions, as decisions based on these outputs can have significant consequences. This study benchmarks methods for assessing pointwise reliability, focusing on data-driven techniques based on the density principle and the local fit principle. These methods evaluate the reliability of individual predictions by analyzing their similarity to training data and evaluating the performance of the model in local regions. Aiming to establish a standardized comparison, the study introduces a benchmark framework that combines error rate evaluations across reliability intervals with t-distributed Stochastic Neighbor Embedding visualizations to further validate the results. The results demonstrate that methods combining density and local fit principles generally outperform those relying on a single principle, achieving lower error rates for high-reliability predictions. Furthermore, the study identifies challenges such as the adjustment of method parameters and clustering limitations and provides insight into their impact on reliability assessments. |
| format | Article |
| id | doaj-art-e089fe7ed6944b6785f786e2f75151d2 |
| institution | OA Journals |
| issn | 2078-2489 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj-art-e089fe7ed6944b6785f786e2f75151d22025-08-20T02:17:59ZengMDPI AGInformation2078-24892025-04-0116432710.3390/info16040327Benchmarking Methods for Pointwise ReliabilityCláudio Correia0Simão Paredes1Teresa Rocha2Jorge Henriques3Jorge Bernardino4Coimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, PortugalCoimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, PortugalCoimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, PortugalCISUC, Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290 Coimbra, PortugalCoimbra Institute of Engineering, Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, PortugalThe growing interest in machine learning in a critical domain like healthcare emphasizes the need for reliable predictions, as decisions based on these outputs can have significant consequences. This study benchmarks methods for assessing pointwise reliability, focusing on data-driven techniques based on the density principle and the local fit principle. These methods evaluate the reliability of individual predictions by analyzing their similarity to training data and evaluating the performance of the model in local regions. Aiming to establish a standardized comparison, the study introduces a benchmark framework that combines error rate evaluations across reliability intervals with t-distributed Stochastic Neighbor Embedding visualizations to further validate the results. The results demonstrate that methods combining density and local fit principles generally outperform those relying on a single principle, achieving lower error rates for high-reliability predictions. Furthermore, the study identifies challenges such as the adjustment of method parameters and clustering limitations and provides insight into their impact on reliability assessments.https://www.mdpi.com/2078-2489/16/4/327benchmarkdensitylocal fitmachine learningpointwise reliability |
| spellingShingle | Cláudio Correia Simão Paredes Teresa Rocha Jorge Henriques Jorge Bernardino Benchmarking Methods for Pointwise Reliability Information benchmark density local fit machine learning pointwise reliability |
| title | Benchmarking Methods for Pointwise Reliability |
| title_full | Benchmarking Methods for Pointwise Reliability |
| title_fullStr | Benchmarking Methods for Pointwise Reliability |
| title_full_unstemmed | Benchmarking Methods for Pointwise Reliability |
| title_short | Benchmarking Methods for Pointwise Reliability |
| title_sort | benchmarking methods for pointwise reliability |
| topic | benchmark density local fit machine learning pointwise reliability |
| url | https://www.mdpi.com/2078-2489/16/4/327 |
| work_keys_str_mv | AT claudiocorreia benchmarkingmethodsforpointwisereliability AT simaoparedes benchmarkingmethodsforpointwisereliability AT teresarocha benchmarkingmethodsforpointwisereliability AT jorgehenriques benchmarkingmethodsforpointwisereliability AT jorgebernardino benchmarkingmethodsforpointwisereliability |