A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning
Abstract Ulcerative colitis (UC) is a chronic inflammatory disorder necessitating precise severity stratification to facilitate optimal therapeutic interventions. This study harnesses a triple-pronged deep learning methodology—including multimodal inference pipelines that eliminate domain-specific t...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Nature Portfolio
2025-07-01
|
| Series: | Scientific Reports |
| Subjects: | |
| Online Access: | https://doi.org/10.1038/s41598-025-12827-5 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849766743719804928 |
|---|---|
| author | Md. Nasim Ahmed Dipta Neogi Muhammad Rafsan Kabir Shafin Rahman Sifat Momen Nabeel Mohammed |
| author_facet | Md. Nasim Ahmed Dipta Neogi Muhammad Rafsan Kabir Shafin Rahman Sifat Momen Nabeel Mohammed |
| author_sort | Md. Nasim Ahmed |
| collection | DOAJ |
| description | Abstract Ulcerative colitis (UC) is a chronic inflammatory disorder necessitating precise severity stratification to facilitate optimal therapeutic interventions. This study harnesses a triple-pronged deep learning methodology—including multimodal inference pipelines that eliminate domain-specific training, few-shot meta-learning, and Vision Transformer (ViT)-based ensembling—to classify UC severity within the HyperKvasir dataset. We systematically evaluate multiple vision transformer architectures, discovering that a Swin-Base model achieves an accuracy of 90%, while a soft-voting ensemble of diverse ViT backbones boosts performance to 93%. In parallel, we leverage multimodal pre-trained frameworks (e.g., CLIP, BLIP, FLAVA) integrated with conventional machine learning algorithms, yielding an accuracy of 83%. To address limited annotated data, we deploy few-shot meta-learning approaches (e.g., Matching Networks), attaining 83% accuracy in a 5-shot context. Furthermore, interpretability is enhanced via SHapley Additive exPlanations (SHAP), which interpret both local and global model behaviors, thereby fostering clinical trust in the model’s inferences. These findings underscore the potential of contemporary representation learning and ensemble strategies for robust UC severity classification, highlighting the pivotal role of model transparency in facilitating medical image analysis. |
| format | Article |
| id | doaj-art-7fe677fab44544a695a796b8cebe366b |
| institution | DOAJ |
| issn | 2045-2322 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | Nature Portfolio |
| record_format | Article |
| series | Scientific Reports |
| spelling | doaj-art-7fe677fab44544a695a796b8cebe366b2025-08-20T03:04:29ZengNature PortfolioScientific Reports2045-23222025-07-0115111510.1038/s41598-025-12827-5A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learningMd. Nasim Ahmed0Dipta Neogi1Muhammad Rafsan Kabir2Shafin Rahman3Sifat Momen4Nabeel Mohammed5Department of Electrical and Computer Engineering, North South UniversityDepartment of Electrical and Computer Engineering, North South UniversityDepartment of Electrical and Computer Engineering, North South UniversityDepartment of Electrical and Computer Engineering, North South UniversityDepartment of Electrical and Computer Engineering, North South UniversityDepartment of Electrical and Computer Engineering, North South UniversityAbstract Ulcerative colitis (UC) is a chronic inflammatory disorder necessitating precise severity stratification to facilitate optimal therapeutic interventions. This study harnesses a triple-pronged deep learning methodology—including multimodal inference pipelines that eliminate domain-specific training, few-shot meta-learning, and Vision Transformer (ViT)-based ensembling—to classify UC severity within the HyperKvasir dataset. We systematically evaluate multiple vision transformer architectures, discovering that a Swin-Base model achieves an accuracy of 90%, while a soft-voting ensemble of diverse ViT backbones boosts performance to 93%. In parallel, we leverage multimodal pre-trained frameworks (e.g., CLIP, BLIP, FLAVA) integrated with conventional machine learning algorithms, yielding an accuracy of 83%. To address limited annotated data, we deploy few-shot meta-learning approaches (e.g., Matching Networks), attaining 83% accuracy in a 5-shot context. Furthermore, interpretability is enhanced via SHapley Additive exPlanations (SHAP), which interpret both local and global model behaviors, thereby fostering clinical trust in the model’s inferences. These findings underscore the potential of contemporary representation learning and ensemble strategies for robust UC severity classification, highlighting the pivotal role of model transparency in facilitating medical image analysis.https://doi.org/10.1038/s41598-025-12827-5Vision transformers (ViT)Multimodal modelsFew-shotMeta-learning |
| spellingShingle | Md. Nasim Ahmed Dipta Neogi Muhammad Rafsan Kabir Shafin Rahman Sifat Momen Nabeel Mohammed A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning Scientific Reports Vision transformers (ViT) Multimodal models Few-shot Meta-learning |
| title | A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning |
| title_full | A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning |
| title_fullStr | A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning |
| title_full_unstemmed | A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning |
| title_short | A triple pronged approach for ulcerative colitis severity classification using multimodal, meta, and transformer based learning |
| title_sort | triple pronged approach for ulcerative colitis severity classification using multimodal meta and transformer based learning |
| topic | Vision transformers (ViT) Multimodal models Few-shot Meta-learning |
| url | https://doi.org/10.1038/s41598-025-12827-5 |
| work_keys_str_mv | AT mdnasimahmed atripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT diptaneogi atripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT muhammadrafsankabir atripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT shafinrahman atripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT sifatmomen atripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT nabeelmohammed atripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT mdnasimahmed tripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT diptaneogi tripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT muhammadrafsankabir tripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT shafinrahman tripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT sifatmomen tripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning AT nabeelmohammed tripleprongedapproachforulcerativecolitisseverityclassificationusingmultimodalmetaandtransformerbasedlearning |