Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews
Abstract classification in systematic reviews (SRs) is a crucial step in evidence synthesis but is often time-consuming and labour-intensive. This study evaluates the effectiveness of various Machine Learning (ML) models and embedding techniques in automating this process. Five diverse datasets are...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10845770/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832576801265680384 |
|---|---|
| author | Mohammad Shariful Islam Mohammad Abu Tareq Rony Md Rasel Hossain Samah Alshathri Walid El-Shafai |
| author_facet | Mohammad Shariful Islam Mohammad Abu Tareq Rony Md Rasel Hossain Samah Alshathri Walid El-Shafai |
| author_sort | Mohammad Shariful Islam |
| collection | DOAJ |
| description | Abstract classification in systematic reviews (SRs) is a crucial step in evidence synthesis but is often time-consuming and labour-intensive. This study evaluates the effectiveness of various Machine Learning (ML) models and embedding techniques in automating this process. Five diverse datasets are utilized: Aceves-Martins (2021), comprising 1,258 excluded and 230 included abstracts on the utilization of animal models in depressive behaviour studies; Bannach-Brown (2016), with 896 excluded and 73 included abstracts focusing on the methodological rigour of environmental health systematic reviews; Meijboom (2021), containing 599 excluded and 32 included abstracts on the retransitioning of Etanercept in rheumatic disease patients; Menon (2022), with 896 excluded and 73 included abstracts on environmental health reviews; and a custom Clinical Review Paper Abstract (CRPA) dataset, featuring 500 excluded and 50 included abstracts. A significant research gap in abstract classification has been identified in previous literature, particularly in comparing Large Language Models (LLMs) with traditional ML and Natural Language Processing (NLP) techniques regarding scalability, adaptability, computational efficiency, and real-time application. Addressing this gap, this study employs GloVe for word embedding via matrix factorization, FastText for character n-gram representation, and Doc2Vec for capturing paragraph-level semantics. A novel Zero-BertXGB technique is introduced, integrating a transformer-based language model, zero-shot learning, and an ML classifier to enhance abstract screening and classification into “Include” or “Exclude” categories. This approach leverages contextual understanding and precision for efficient abstract processing. The Zero-BertXGB technique is compared against other prominent LLMs, including BERT, PaLM, LLaMA, GPT-3.5, and GPT-4, to validate its effectiveness. The Zero-BertXGB model achieved accuracy values of 99.3% for Aceves-Martins2021, 92.6% for Bannach-Brown2016, 85.7% for Meijboom2021, 94.1% for Menon2022, and 98.8% for CRPA. The findings indicate that the Zero-BertXGB model, alongside other LLMs, can deliver reliable results with minimal human intervention, enhancing abstract screening efficiency and potentially revolutionizing systematic review workflows. |
| format | Article |
| id | doaj-art-fdd7f32c339d412d85e52932f6a31647 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-fdd7f32c339d412d85e52932f6a316472025-01-31T00:00:56ZengIEEEIEEE Access2169-35362025-01-0113184181844010.1109/ACCESS.2025.353177810845770Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic ReviewsMohammad Shariful Islam0https://orcid.org/0009-0007-8442-1425Mohammad Abu Tareq Rony1https://orcid.org/0000-0002-0640-1425Md Rasel Hossain2Samah Alshathri3https://orcid.org/0000-0002-8805-7890Walid El-Shafai4https://orcid.org/0000-0001-7509-2120Department of Computer Science and Telecommunication Engineering, Noakhali Science and Technology University, Noakhali, BangladeshDepartment of Statistics, Noakhali Science and Technology University, Noakhali, BangladeshDepartment of Statistics, Noakhali Science and Technology University, Noakhali, BangladeshDepartment of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box, 84428, Riyadh, Saudi ArabiaComputer Science Department, Automated Systems and Soft Computing Laboratory (ASSCL), Prince Sultan University, Riyadh, Saudi ArabiaAbstract classification in systematic reviews (SRs) is a crucial step in evidence synthesis but is often time-consuming and labour-intensive. This study evaluates the effectiveness of various Machine Learning (ML) models and embedding techniques in automating this process. Five diverse datasets are utilized: Aceves-Martins (2021), comprising 1,258 excluded and 230 included abstracts on the utilization of animal models in depressive behaviour studies; Bannach-Brown (2016), with 896 excluded and 73 included abstracts focusing on the methodological rigour of environmental health systematic reviews; Meijboom (2021), containing 599 excluded and 32 included abstracts on the retransitioning of Etanercept in rheumatic disease patients; Menon (2022), with 896 excluded and 73 included abstracts on environmental health reviews; and a custom Clinical Review Paper Abstract (CRPA) dataset, featuring 500 excluded and 50 included abstracts. A significant research gap in abstract classification has been identified in previous literature, particularly in comparing Large Language Models (LLMs) with traditional ML and Natural Language Processing (NLP) techniques regarding scalability, adaptability, computational efficiency, and real-time application. Addressing this gap, this study employs GloVe for word embedding via matrix factorization, FastText for character n-gram representation, and Doc2Vec for capturing paragraph-level semantics. A novel Zero-BertXGB technique is introduced, integrating a transformer-based language model, zero-shot learning, and an ML classifier to enhance abstract screening and classification into “Include” or “Exclude” categories. This approach leverages contextual understanding and precision for efficient abstract processing. The Zero-BertXGB technique is compared against other prominent LLMs, including BERT, PaLM, LLaMA, GPT-3.5, and GPT-4, to validate its effectiveness. The Zero-BertXGB model achieved accuracy values of 99.3% for Aceves-Martins2021, 92.6% for Bannach-Brown2016, 85.7% for Meijboom2021, 94.1% for Menon2022, and 98.8% for CRPA. The findings indicate that the Zero-BertXGB model, alongside other LLMs, can deliver reliable results with minimal human intervention, enhancing abstract screening efficiency and potentially revolutionizing systematic review workflows.https://ieeexplore.ieee.org/document/10845770/Abstract classificationmachine learningnatural language processingzero-BertXGB methodslarge language models |
| spellingShingle | Mohammad Shariful Islam Mohammad Abu Tareq Rony Md Rasel Hossain Samah Alshathri Walid El-Shafai Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews IEEE Access Abstract classification machine learning natural language processing zero-BertXGB methods large language models |
| title | Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews |
| title_full | Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews |
| title_fullStr | Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews |
| title_full_unstemmed | Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews |
| title_short | Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews |
| title_sort | zero bertxgb an empirical technique for abstract classification in systematic reviews |
| topic | Abstract classification machine learning natural language processing zero-BertXGB methods large language models |
| url | https://ieeexplore.ieee.org/document/10845770/ |
| work_keys_str_mv | AT mohammadsharifulislam zerobertxgbanempiricaltechniqueforabstractclassificationinsystematicreviews AT mohammadabutareqrony zerobertxgbanempiricaltechniqueforabstractclassificationinsystematicreviews AT mdraselhossain zerobertxgbanempiricaltechniqueforabstractclassificationinsystematicreviews AT samahalshathri zerobertxgbanempiricaltechniqueforabstractclassificationinsystematicreviews AT walidelshafai zerobertxgbanempiricaltechniqueforabstractclassificationinsystematicreviews |