Practical classification accuracy of sequential data using neural networks
Many existing studies on neural network accuracy utilize datasets that may not always reflect real-world conditions. While it has been demonstrated that accuracy tends to decrease as the number of benign samples increases, this effect has not been quantitatively assessed within neural networks. More...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2025-03-01
|
| Series: | Machine Learning with Applications |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666827024000872 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850077277666148352 |
|---|---|
| author | Mamoru Mimura |
| author_facet | Mamoru Mimura |
| author_sort | Mamoru Mimura |
| collection | DOAJ |
| description | Many existing studies on neural network accuracy utilize datasets that may not always reflect real-world conditions. While it has been demonstrated that accuracy tends to decrease as the number of benign samples increases, this effect has not been quantitatively assessed within neural networks. Moreover, its relevance to security tasks beyond malware classification remains unexplored. In this research, we refined the metric to evaluate the degradation of accuracy with an increased number of benign samples in test data. Utilizing both standard and specific neural network models, we conducted experiments to adapt this metric to neural networks and various feature extraction techniques. Using the FFRI dataset, comprising 150,000 malware and 400,000 benign samples, along with the URL dataset, containing 3143 malicious and 106,545,781 benign samples, we increased benign samples in the test set while keeping the training set’s malicious and benign samples constant. Our findings indicate that neural networks can indeed overestimate their accuracy with a smaller count of benign samples. Importantly, our refined metric is not only applicable to neural networks but is also effective for other feature extraction methods and security tasks beyond malware detection. |
| format | Article |
| id | doaj-art-8da9124bdc284224bc3a6c7667b3b092 |
| institution | DOAJ |
| issn | 2666-8270 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Machine Learning with Applications |
| spelling | doaj-art-8da9124bdc284224bc3a6c7667b3b0922025-08-20T02:45:50ZengElsevierMachine Learning with Applications2666-82702025-03-011910061110.1016/j.mlwa.2024.100611Practical classification accuracy of sequential data using neural networksMamoru Mimura0National Defense Academy, 1-10-20 Hashirimizu, Yokosuka, Kanagawa, JapanMany existing studies on neural network accuracy utilize datasets that may not always reflect real-world conditions. While it has been demonstrated that accuracy tends to decrease as the number of benign samples increases, this effect has not been quantitatively assessed within neural networks. Moreover, its relevance to security tasks beyond malware classification remains unexplored. In this research, we refined the metric to evaluate the degradation of accuracy with an increased number of benign samples in test data. Utilizing both standard and specific neural network models, we conducted experiments to adapt this metric to neural networks and various feature extraction techniques. Using the FFRI dataset, comprising 150,000 malware and 400,000 benign samples, along with the URL dataset, containing 3143 malicious and 106,545,781 benign samples, we increased benign samples in the test set while keeping the training set’s malicious and benign samples constant. Our findings indicate that neural networks can indeed overestimate their accuracy with a smaller count of benign samples. Importantly, our refined metric is not only applicable to neural networks but is also effective for other feature extraction methods and security tasks beyond malware detection.http://www.sciencedirect.com/science/article/pii/S2666827024000872Machine learningBinary classificationBenign sampleDeep neural networkRecurrent neural networkLong short-term memory |
| spellingShingle | Mamoru Mimura Practical classification accuracy of sequential data using neural networks Machine Learning with Applications Machine learning Binary classification Benign sample Deep neural network Recurrent neural network Long short-term memory |
| title | Practical classification accuracy of sequential data using neural networks |
| title_full | Practical classification accuracy of sequential data using neural networks |
| title_fullStr | Practical classification accuracy of sequential data using neural networks |
| title_full_unstemmed | Practical classification accuracy of sequential data using neural networks |
| title_short | Practical classification accuracy of sequential data using neural networks |
| title_sort | practical classification accuracy of sequential data using neural networks |
| topic | Machine learning Binary classification Benign sample Deep neural network Recurrent neural network Long short-term memory |
| url | http://www.sciencedirect.com/science/article/pii/S2666827024000872 |
| work_keys_str_mv | AT mamorumimura practicalclassificationaccuracyofsequentialdatausingneuralnetworks |