Calibrating AI Models for Wireless Communications via Conformal Prediction
When used in complex engineered systems, such as communication networks, artificial intelligence (AI) models should be not only as accurate as possible, but also well calibrated. A well-calibrated AI model is one that can reliably quantify the uncertainty of its decisions, assigning high confidence...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2023-01-01
|
| Series: | IEEE Transactions on Machine Learning in Communications and Networking |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10262367/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850035968017432576 |
|---|---|
| author | Kfir M. Cohen Sangwoo Park Osvaldo Simeone Shlomo Shamai Shitz |
| author_facet | Kfir M. Cohen Sangwoo Park Osvaldo Simeone Shlomo Shamai Shitz |
| author_sort | Kfir M. Cohen |
| collection | DOAJ |
| description | When used in complex engineered systems, such as communication networks, artificial intelligence (AI) models should be not only as accurate as possible, but also well calibrated. A well-calibrated AI model is one that can reliably quantify the uncertainty of its decisions, assigning high confidence levels to decisions that are likely to be correct, and low confidence levels to decisions that are likely to be erroneous. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees. Conformal prediction transforms probabilistic predictors into set predictors that are guaranteed to contain the correct answer with a probability chosen by the designer. Such formal calibration guarantees hold irrespective of the true, unknown, distribution underlying the generation of the variables of interest, and can be defined in terms of ensemble or time-averaged probabilities. In this paper, conformal prediction is applied for the first time to the design of AI for communication systems in conjunction with both frequentist and Bayesian learning, focusing on the key tasks of demodulation, modulation classification, and channel prediction. For demodulation and modulation classification, we apply both validation-based and cross-validation-based conformal prediction; while we investigate the use of online conformal prediction for channel prediction. For each task, we evaluate the probability that the set predictor contains the true output, validating the theoretical coverage guarantees of conformal prediction, as well as the informativeness of the predictor via the average predicted set size. |
| format | Article |
| id | doaj-art-2a5a3211a9d043e4b38bea8d6fdcd995 |
| institution | DOAJ |
| issn | 2831-316X |
| language | English |
| publishDate | 2023-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Transactions on Machine Learning in Communications and Networking |
| spelling | doaj-art-2a5a3211a9d043e4b38bea8d6fdcd9952025-08-20T02:57:19ZengIEEEIEEE Transactions on Machine Learning in Communications and Networking2831-316X2023-01-01129631210.1109/TMLCN.2023.331928210262367Calibrating AI Models for Wireless Communications via Conformal PredictionKfir M. Cohen0https://orcid.org/0000-0001-8041-3873Sangwoo Park1https://orcid.org/0000-0003-4091-7860Osvaldo Simeone2https://orcid.org/0000-0001-9898-3209Shlomo Shamai Shitz3https://orcid.org/0000-0002-6594-3371Department of Engineering, King’s Communication, Learning and Information Processing (KCLIP) Laboratory, King’s College London, London, U.K.Department of Engineering, King’s Communication, Learning and Information Processing (KCLIP) Laboratory, King’s College London, London, U.K.Department of Engineering, King’s Communication, Learning and Information Processing (KCLIP) Laboratory, King’s College London, London, U.K.Viterbi Faculty of Electrical and Computing Engineering, Technion—Israel Institute of Technology, Haifa, IsraelWhen used in complex engineered systems, such as communication networks, artificial intelligence (AI) models should be not only as accurate as possible, but also well calibrated. A well-calibrated AI model is one that can reliably quantify the uncertainty of its decisions, assigning high confidence levels to decisions that are likely to be correct, and low confidence levels to decisions that are likely to be erroneous. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees. Conformal prediction transforms probabilistic predictors into set predictors that are guaranteed to contain the correct answer with a probability chosen by the designer. Such formal calibration guarantees hold irrespective of the true, unknown, distribution underlying the generation of the variables of interest, and can be defined in terms of ensemble or time-averaged probabilities. In this paper, conformal prediction is applied for the first time to the design of AI for communication systems in conjunction with both frequentist and Bayesian learning, focusing on the key tasks of demodulation, modulation classification, and channel prediction. For demodulation and modulation classification, we apply both validation-based and cross-validation-based conformal prediction; while we investigate the use of online conformal prediction for channel prediction. For each task, we evaluate the probability that the set predictor contains the true output, validating the theoretical coverage guarantees of conformal prediction, as well as the informativeness of the predictor via the average predicted set size.https://ieeexplore.ieee.org/document/10262367/Bayesian learningcalibrationconformal predictioncross-validationreliabilityset prediction |
| spellingShingle | Kfir M. Cohen Sangwoo Park Osvaldo Simeone Shlomo Shamai Shitz Calibrating AI Models for Wireless Communications via Conformal Prediction IEEE Transactions on Machine Learning in Communications and Networking Bayesian learning calibration conformal prediction cross-validation reliability set prediction |
| title | Calibrating AI Models for Wireless Communications via Conformal Prediction |
| title_full | Calibrating AI Models for Wireless Communications via Conformal Prediction |
| title_fullStr | Calibrating AI Models for Wireless Communications via Conformal Prediction |
| title_full_unstemmed | Calibrating AI Models for Wireless Communications via Conformal Prediction |
| title_short | Calibrating AI Models for Wireless Communications via Conformal Prediction |
| title_sort | calibrating ai models for wireless communications via conformal prediction |
| topic | Bayesian learning calibration conformal prediction cross-validation reliability set prediction |
| url | https://ieeexplore.ieee.org/document/10262367/ |
| work_keys_str_mv | AT kfirmcohen calibratingaimodelsforwirelesscommunicationsviaconformalprediction AT sangwoopark calibratingaimodelsforwirelesscommunicationsviaconformalprediction AT osvaldosimeone calibratingaimodelsforwirelesscommunicationsviaconformalprediction AT shlomoshamaishitz calibratingaimodelsforwirelesscommunicationsviaconformalprediction |