Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models
The performance of software defect prediction (SDP) models determines the priority of test resource allocation. Researchers also use interpretability techniques to gain empirical knowledge about software quality from SDP models. However, SDP methods designed in the past research rarely consider the...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Wiley
2023-01-01
|
| Series: | IET Software |
| Online Access: | http://dx.doi.org/10.1049/2023/6293074 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832548024863162368 |
|---|---|
| author | Yu Zhao Zhiqiu Huang Lina Gong Yi Zhu Qiao Yu Yuxiang Gao |
| author_facet | Yu Zhao Zhiqiu Huang Lina Gong Yi Zhu Qiao Yu Yuxiang Gao |
| author_sort | Yu Zhao |
| collection | DOAJ |
| description | The performance of software defect prediction (SDP) models determines the priority of test resource allocation. Researchers also use interpretability techniques to gain empirical knowledge about software quality from SDP models. However, SDP methods designed in the past research rarely consider the impact of data transformation methods, simple but commonly used preprocessing techniques, on the performance and interpretability of SDP models. Therefore, in this paper, we investigate the impact of three data transformation methods (Log, Minmax, and Z-score) on the performance and interpretability of SDP models. Through empirical research on (i) six classification techniques (random forest, decision tree, logistic regression, Naive Bayes, K-nearest neighbors, and multilayer perceptron), (ii) six performance evaluation indicators (Accuracy, Precision, Recall, F1, MCC, and AUC), (iii) two interpretable methods (permutation and SHAP), (iv) two feature importance measures (Top-k feature rank overlap and difference), and (v) three datasets (Promise, Relink, and AEEEM), our results show that the data transformation methods can significantly improve the performance of the SDP models and greatly affect the variation of the most important features. Specifically, the impact of data transformation methods on the performance and interpretability of SDP models depends on the classification techniques and evaluation indicators. We observe that log transformation improves NB model performance by 7%–61% on the other five indicators with a 5% drop in Precision. Minmax and Z-score transformation improves NB model performance by 2%–9% across all indicators. However, all three transformation methods lead to substantial changes in the Top-5 important feature ranks, with differences exceeding 2 in 40%–80% of cases (detailed results available in the main content). Based on our findings, we recommend that (1) considering the impact of data transformation methods on model performance and interpretability when designing SDP approaches as transformations can improve model accuracy, and potentially obscure important features, which lead to challenges in interpretation, (2) conducting comparative experiments with and without the transformations to validate the effectiveness of proposed methods which are designed to improve the prediction performance, and (3) tracking changes in the most important features before and after applying data transformation methods to ensure precise and traceable interpretability conclusions to gain insights. Our study reminds researchers and practitioners of the need for comprehensive considerations even when using other similar simple data processing methods. |
| format | Article |
| id | doaj-art-ec1ea11b58d344d69383787df69c6881 |
| institution | Kabale University |
| issn | 1751-8814 |
| language | English |
| publishDate | 2023-01-01 |
| publisher | Wiley |
| record_format | Article |
| series | IET Software |
| spelling | doaj-art-ec1ea11b58d344d69383787df69c68812025-02-03T06:42:46ZengWileyIET Software1751-88142023-01-01202310.1049/2023/6293074Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction ModelsYu Zhao0Zhiqiu Huang1Lina Gong2Yi Zhu3Qiao Yu4Yuxiang Gao5School of Computer Science and TechnologySchool of Computer Science and TechnologySchool of Computer Science and TechnologySchool of Computer Science and TechnologySchool of Computer Science and TechnologySchool of Computer Science and TechnologyThe performance of software defect prediction (SDP) models determines the priority of test resource allocation. Researchers also use interpretability techniques to gain empirical knowledge about software quality from SDP models. However, SDP methods designed in the past research rarely consider the impact of data transformation methods, simple but commonly used preprocessing techniques, on the performance and interpretability of SDP models. Therefore, in this paper, we investigate the impact of three data transformation methods (Log, Minmax, and Z-score) on the performance and interpretability of SDP models. Through empirical research on (i) six classification techniques (random forest, decision tree, logistic regression, Naive Bayes, K-nearest neighbors, and multilayer perceptron), (ii) six performance evaluation indicators (Accuracy, Precision, Recall, F1, MCC, and AUC), (iii) two interpretable methods (permutation and SHAP), (iv) two feature importance measures (Top-k feature rank overlap and difference), and (v) three datasets (Promise, Relink, and AEEEM), our results show that the data transformation methods can significantly improve the performance of the SDP models and greatly affect the variation of the most important features. Specifically, the impact of data transformation methods on the performance and interpretability of SDP models depends on the classification techniques and evaluation indicators. We observe that log transformation improves NB model performance by 7%–61% on the other five indicators with a 5% drop in Precision. Minmax and Z-score transformation improves NB model performance by 2%–9% across all indicators. However, all three transformation methods lead to substantial changes in the Top-5 important feature ranks, with differences exceeding 2 in 40%–80% of cases (detailed results available in the main content). Based on our findings, we recommend that (1) considering the impact of data transformation methods on model performance and interpretability when designing SDP approaches as transformations can improve model accuracy, and potentially obscure important features, which lead to challenges in interpretation, (2) conducting comparative experiments with and without the transformations to validate the effectiveness of proposed methods which are designed to improve the prediction performance, and (3) tracking changes in the most important features before and after applying data transformation methods to ensure precise and traceable interpretability conclusions to gain insights. Our study reminds researchers and practitioners of the need for comprehensive considerations even when using other similar simple data processing methods.http://dx.doi.org/10.1049/2023/6293074 |
| spellingShingle | Yu Zhao Zhiqiu Huang Lina Gong Yi Zhu Qiao Yu Yuxiang Gao Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models IET Software |
| title | Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models |
| title_full | Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models |
| title_fullStr | Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models |
| title_full_unstemmed | Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models |
| title_short | Evaluating the Impact of Data Transformation Techniques on the Performance and Interpretability of Software Defect Prediction Models |
| title_sort | evaluating the impact of data transformation techniques on the performance and interpretability of software defect prediction models |
| url | http://dx.doi.org/10.1049/2023/6293074 |
| work_keys_str_mv | AT yuzhao evaluatingtheimpactofdatatransformationtechniquesontheperformanceandinterpretabilityofsoftwaredefectpredictionmodels AT zhiqiuhuang evaluatingtheimpactofdatatransformationtechniquesontheperformanceandinterpretabilityofsoftwaredefectpredictionmodels AT linagong evaluatingtheimpactofdatatransformationtechniquesontheperformanceandinterpretabilityofsoftwaredefectpredictionmodels AT yizhu evaluatingtheimpactofdatatransformationtechniquesontheperformanceandinterpretabilityofsoftwaredefectpredictionmodels AT qiaoyu evaluatingtheimpactofdatatransformationtechniquesontheperformanceandinterpretabilityofsoftwaredefectpredictionmodels AT yuxianggao evaluatingtheimpactofdatatransformationtechniquesontheperformanceandinterpretabilityofsoftwaredefectpredictionmodels |