Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers
The rapid increase in the number of medical image repositories nowadays has led to problems in managing and retrieving medical visual data. This has proved the necessity of Content-Based Image Retrieval (CBIR) with the aim of facilitating the investigation of such medical imagery. One of the most se...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
OICC Press
2022-09-01
|
| Series: | Majlesi Journal of Electrical Engineering |
| Subjects: | |
| Online Access: | https://oiccpress.com/mjee/article/view/4956 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850283557557108736 |
|---|---|
| author | Ahmed Yahya Dalya Khaled Waleed Al-Azzawi Tawfeeq Alghazali H. Sabah Jabr R. Madhat Abdulla M. Kadhim Abbas Al-Maeeni N. Hussin Alwan S. Saad Najeeb Kh. T. Falih |
| author_facet | Ahmed Yahya Dalya Khaled Waleed Al-Azzawi Tawfeeq Alghazali H. Sabah Jabr R. Madhat Abdulla M. Kadhim Abbas Al-Maeeni N. Hussin Alwan S. Saad Najeeb Kh. T. Falih |
| author_sort | Ahmed Yahya |
| collection | DOAJ |
| description | The rapid increase in the number of medical image repositories nowadays has led to problems in managing and retrieving medical visual data. This has proved the necessity of Content-Based Image Retrieval (CBIR) with the aim of facilitating the investigation of such medical imagery. One of the most serious challenges that require special attention is the representational quality of the embeddings generated by the retrieval pipelines. These embeddings should include global and local features to obtain more useful information from the input data. To fill this gap, in this paper, we propose a CBIR framework that utilizes the power of deep neural networks to efficiently classify and fetch the most related medical images with respect to a query image. Our proposed model is based on combining Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) and learns to capture both the locality and also the globality of high-level feature maps. Our method is trained to encode the images in the database and outputs a ranking list containing the most similar image to the least similar one to the query. To conduct our experiments, an intermodal dataset containing ten classes with five different modalities is used to train and assess the proposed framework. The results show an average classification accuracy of 95.32 % and a mean average precision of 0.61. Our proposed framework can be very effective in retrieving multimodal medical images with the images of different organs in the body. |
| format | Article |
| id | doaj-art-2e57c121452b4654b15e5341dd4c2749 |
| institution | OA Journals |
| issn | 2345-377X 2345-3796 |
| language | English |
| publishDate | 2022-09-01 |
| publisher | OICC Press |
| record_format | Article |
| series | Majlesi Journal of Electrical Engineering |
| spelling | doaj-art-2e57c121452b4654b15e5341dd4c27492025-08-20T01:47:45ZengOICC PressMajlesi Journal of Electrical Engineering2345-377X2345-37962022-09-0116310.30486/mjee.2022.696500Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision TransformersAhmed Yahya0Dalya Khaled1Waleed Al-Azzawi2Tawfeeq Alghazali3H. Sabah Jabr4R. Madhat Abdulla5M. Kadhim Abbas Al-Maeeni6N. Hussin Alwan7S. Saad Najeeb8Kh. T. Falih9Department of nursing, Al-Hadba University College, IraqAl-Manara College for Medical Sciences, Maysan, IraqMedical Lab. Techniques department, College of Medical Technology, Al-Farahidi University, IraqCollege of Media, Department of Journalism, The Islamic University in Najaf, Najaf, IraqAnesthesia Techniques Department, Al-Mustaqbal University College, Babylon, IraqThe University of Mashreq, Baghdad, IraqAl-Nisour University College, Baghdad, IraqDepartment of Nursing, Al-Zahrawi University College, Karbala, IraqAl-Esraa University College, Baghdad, IraqNew Era and Development in Civil Engineering Research Group, Scientific Research Center, Al-Ayen University, Thi-Qar, Iraq.The rapid increase in the number of medical image repositories nowadays has led to problems in managing and retrieving medical visual data. This has proved the necessity of Content-Based Image Retrieval (CBIR) with the aim of facilitating the investigation of such medical imagery. One of the most serious challenges that require special attention is the representational quality of the embeddings generated by the retrieval pipelines. These embeddings should include global and local features to obtain more useful information from the input data. To fill this gap, in this paper, we propose a CBIR framework that utilizes the power of deep neural networks to efficiently classify and fetch the most related medical images with respect to a query image. Our proposed model is based on combining Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) and learns to capture both the locality and also the globality of high-level feature maps. Our method is trained to encode the images in the database and outputs a ranking list containing the most similar image to the least similar one to the query. To conduct our experiments, an intermodal dataset containing ten classes with five different modalities is used to train and assess the proposed framework. The results show an average classification accuracy of 95.32 % and a mean average precision of 0.61. Our proposed framework can be very effective in retrieving multimodal medical images with the images of different organs in the body.https://oiccpress.com/mjee/article/view/4956Content-based image retrievalConvolutional neural networksEnsemble Learningmedical image retrievalsimilarity-based visual searchvision transformers. deep learning |
| spellingShingle | Ahmed Yahya Dalya Khaled Waleed Al-Azzawi Tawfeeq Alghazali H. Sabah Jabr R. Madhat Abdulla M. Kadhim Abbas Al-Maeeni N. Hussin Alwan S. Saad Najeeb Kh. T. Falih Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers Majlesi Journal of Electrical Engineering Content-based image retrieval Convolutional neural networks Ensemble Learning medical image retrieval similarity-based visual search vision transformers. deep learning |
| title | Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers |
| title_full | Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers |
| title_fullStr | Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers |
| title_full_unstemmed | Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers |
| title_short | Medical Image Retrieval Based on Ensemble Learning using Convolutional Neural Networks and Vision Transformers |
| title_sort | medical image retrieval based on ensemble learning using convolutional neural networks and vision transformers |
| topic | Content-based image retrieval Convolutional neural networks Ensemble Learning medical image retrieval similarity-based visual search vision transformers. deep learning |
| url | https://oiccpress.com/mjee/article/view/4956 |
| work_keys_str_mv | AT ahmedyahya medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT dalyakhaled medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT waleedalazzawi medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT tawfeeqalghazali medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT hsabahjabr medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT rmadhatabdulla medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT mkadhimabbasalmaeeni medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT nhussinalwan medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT ssaadnajeeb medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers AT khtfalih medicalimageretrievalbasedonensemblelearningusingconvolutionalneuralnetworksandvisiontransformers |