Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability
Regression is a fundamental task in machine learning, and neural networks have been successfully employed in many applications to identify underlying regression patterns. However, they are often criticised for their lack of interpretability and commonly referred to as black-box models. Feature selec...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-04-01
|
| Series: | Machine Learning and Knowledge Extraction |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2504-4990/7/2/33 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849431828269629440 |
|---|---|
| author | Georgios I. Liapis Sophia Tsoka Lazaros G. Papageorgiou |
| author_facet | Georgios I. Liapis Sophia Tsoka Lazaros G. Papageorgiou |
| author_sort | Georgios I. Liapis |
| collection | DOAJ |
| description | Regression is a fundamental task in machine learning, and neural networks have been successfully employed in many applications to identify underlying regression patterns. However, they are often criticised for their lack of interpretability and commonly referred to as black-box models. Feature selection approaches address this challenge by simplifying datasets through the removal of unimportant features, while improving explainability by revealing feature importance. In this work, we leverage mathematical programming to identify the most important features in a trained deep neural network with a ReLU activation function, providing greater insight into its decision-making process. Unlike traditional feature selection methods, our approach adjusts the weights and biases of the trained neural network via a Mixed-Integer Linear Programming (MILP) model to identify the most important features and thereby uncover underlying relationships. The mathematical formulation is reported, which determines the subset of selected features, and clustering is applied to reduce the complexity of the model. Our results illustrate improved performance in the neural network when feature selection is implemented by the proposed approach, as compared to other feature selection approaches. Finally, analysis of feature selection frequency across each dataset reveals feature contribution in model predictions, thereby addressing the black-box nature of the neural network. |
| format | Article |
| id | doaj-art-2207416f1b07425b99fa4b2559971eaf |
| institution | Kabale University |
| issn | 2504-4990 |
| language | English |
| publishDate | 2025-04-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Machine Learning and Knowledge Extraction |
| spelling | doaj-art-2207416f1b07425b99fa4b2559971eaf2025-08-20T03:27:30ZengMDPI AGMachine Learning and Knowledge Extraction2504-49902025-04-01723310.3390/make7020033Optimisation-Based Feature Selection for Regression Neural Networks Towards ExplainabilityGeorgios I. Liapis0Sophia Tsoka1Lazaros G. Papageorgiou2The Sargent Centre for Process Systems Engineering, Department of Chemical Engineering, UCL (University College London), Torrington Place, London WC1E 7JE, UKDepartment of Informatics, King’s College London, Strand, London WC2R 2LS, UKThe Sargent Centre for Process Systems Engineering, Department of Chemical Engineering, UCL (University College London), Torrington Place, London WC1E 7JE, UKRegression is a fundamental task in machine learning, and neural networks have been successfully employed in many applications to identify underlying regression patterns. However, they are often criticised for their lack of interpretability and commonly referred to as black-box models. Feature selection approaches address this challenge by simplifying datasets through the removal of unimportant features, while improving explainability by revealing feature importance. In this work, we leverage mathematical programming to identify the most important features in a trained deep neural network with a ReLU activation function, providing greater insight into its decision-making process. Unlike traditional feature selection methods, our approach adjusts the weights and biases of the trained neural network via a Mixed-Integer Linear Programming (MILP) model to identify the most important features and thereby uncover underlying relationships. The mathematical formulation is reported, which determines the subset of selected features, and clustering is applied to reduce the complexity of the model. Our results illustrate improved performance in the neural network when feature selection is implemented by the proposed approach, as compared to other feature selection approaches. Finally, analysis of feature selection frequency across each dataset reveals feature contribution in model predictions, thereby addressing the black-box nature of the neural network.https://www.mdpi.com/2504-4990/7/2/33mathematical programmingneural networkmixed-integer optimisationfeature selectionexplainable machine learning |
| spellingShingle | Georgios I. Liapis Sophia Tsoka Lazaros G. Papageorgiou Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability Machine Learning and Knowledge Extraction mathematical programming neural network mixed-integer optimisation feature selection explainable machine learning |
| title | Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability |
| title_full | Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability |
| title_fullStr | Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability |
| title_full_unstemmed | Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability |
| title_short | Optimisation-Based Feature Selection for Regression Neural Networks Towards Explainability |
| title_sort | optimisation based feature selection for regression neural networks towards explainability |
| topic | mathematical programming neural network mixed-integer optimisation feature selection explainable machine learning |
| url | https://www.mdpi.com/2504-4990/7/2/33 |
| work_keys_str_mv | AT georgiosiliapis optimisationbasedfeatureselectionforregressionneuralnetworkstowardsexplainability AT sophiatsoka optimisationbasedfeatureselectionforregressionneuralnetworkstowardsexplainability AT lazarosgpapageorgiou optimisationbasedfeatureselectionforregressionneuralnetworkstowardsexplainability |