Conversational LLM-Based Decision Support for Defect Classification in AFM Images
Atomic force microscopy (AFM) has emerged as a powerful tool for nanoscale imaging and quantitative characterization of organic (e.g., live cells, proteins, DNA, and lipid bilayers) and inorganic (e.g., silicon wafers and polymers) specimens. However, image artifacts in AFM height and peak force err...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Open Journal of Instrumentation and Measurement |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11096088/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849228533406105600 |
|---|---|
| author | Angona Biswas Jaydeep Rade Nabila Masud Md Hasibul Hasan Hasib Aditya Balu Juntao Zhang Soumik Sarkar Adarsh Krishnamurthy Juan Ren Anwesha Sarkar |
| author_facet | Angona Biswas Jaydeep Rade Nabila Masud Md Hasibul Hasan Hasib Aditya Balu Juntao Zhang Soumik Sarkar Adarsh Krishnamurthy Juan Ren Anwesha Sarkar |
| author_sort | Angona Biswas |
| collection | DOAJ |
| description | Atomic force microscopy (AFM) has emerged as a powerful tool for nanoscale imaging and quantitative characterization of organic (e.g., live cells, proteins, DNA, and lipid bilayers) and inorganic (e.g., silicon wafers and polymers) specimens. However, image artifacts in AFM height and peak force error images directly affect the precision of nanomechanical measurements. Experimentalists face considerable challenges in obtaining high-quality AFM images due to the requirement of specialized expertise and constant manual monitoring. Another challenge is the lack of high-quality AFM datasets to train machine learning models for automated defect detection. In this work, we propose a two-step AI framework that combines a vision-based deep learning (DL) model for classifying AFM image defects with a large language model (LLM)-based conversational assistant that provides real-time corrective guidance in natural language, making it particularly valuable for non-AFM experts aiming to obtain high-quality images. We curated an annotated AFM defect dataset spanning organic and inorganic samples to train the defect detection model. Our defect classification model achieves 91.43% overall accuracy, with a recall of 93% for tip contamination and 60% not-tracking defects. We further develop an intuitive user interface that enables seamless interaction with the DL model and integrates an LLM-based guidance feature to support users in understanding defects and improving future experiments. We then evaluate the performance of multiple state-of-the-art LLMs on AFM-related queries, offering users flexibility in LLM selection based on their specific needs. LLM evaluations and the benchmark questions are available at: <uri>https://github.com/idealab-isu/AFM-LLM-Defect-Guidance</uri>. |
| format | Article |
| id | doaj-art-af19782eb8dd437e9c54c65bd353fb93 |
| institution | Kabale University |
| issn | 2768-7236 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Open Journal of Instrumentation and Measurement |
| spelling | doaj-art-af19782eb8dd437e9c54c65bd353fb932025-08-22T23:17:35ZengIEEEIEEE Open Journal of Instrumentation and Measurement2768-72362025-01-01411210.1109/OJIM.2025.359228411096088Conversational LLM-Based Decision Support for Defect Classification in AFM ImagesAngona Biswas0https://orcid.org/0009-0004-8916-0728Jaydeep Rade1https://orcid.org/0000-0002-6831-8416Nabila Masud2https://orcid.org/0009-0006-0125-4055Md Hasibul Hasan Hasib3https://orcid.org/0000-0001-5132-5161Aditya Balu4https://orcid.org/0000-0003-2005-2548Juntao Zhang5https://orcid.org/0009-0007-7913-4261Soumik Sarkar6https://orcid.org/0000-0002-6775-9199Adarsh Krishnamurthy7https://orcid.org/0000-0002-5900-1863Juan Ren8https://orcid.org/0000-0002-5616-7219Anwesha Sarkar9https://orcid.org/0000-0002-4267-7242Department of Electrical and Computer Engineering, Iowa State University, Ames, IA, USADepartment of Electrical and Computer Engineering, Iowa State University, Ames, IA, USADepartment of Electrical and Computer Engineering, Iowa State University, Ames, IA, USADepartment of Electrical and Computer Engineering, Iowa State University, Ames, IA, USATranslational AI Center, Iowa State University, Ames, IA, USADepartment of Mechanical Engineering, Iowa State University, Ames, IA, USADepartment of Mechanical Engineering, Iowa State University, Ames, IA, USADepartment of Electrical and Computer Engineering, Iowa State University, Ames, IA, USADepartment of Mechanical Engineering, Iowa State University, Ames, IA, USADepartment of Electrical and Computer Engineering, Iowa State University, Ames, IA, USAAtomic force microscopy (AFM) has emerged as a powerful tool for nanoscale imaging and quantitative characterization of organic (e.g., live cells, proteins, DNA, and lipid bilayers) and inorganic (e.g., silicon wafers and polymers) specimens. However, image artifacts in AFM height and peak force error images directly affect the precision of nanomechanical measurements. Experimentalists face considerable challenges in obtaining high-quality AFM images due to the requirement of specialized expertise and constant manual monitoring. Another challenge is the lack of high-quality AFM datasets to train machine learning models for automated defect detection. In this work, we propose a two-step AI framework that combines a vision-based deep learning (DL) model for classifying AFM image defects with a large language model (LLM)-based conversational assistant that provides real-time corrective guidance in natural language, making it particularly valuable for non-AFM experts aiming to obtain high-quality images. We curated an annotated AFM defect dataset spanning organic and inorganic samples to train the defect detection model. Our defect classification model achieves 91.43% overall accuracy, with a recall of 93% for tip contamination and 60% not-tracking defects. We further develop an intuitive user interface that enables seamless interaction with the DL model and integrates an LLM-based guidance feature to support users in understanding defects and improving future experiments. We then evaluate the performance of multiple state-of-the-art LLMs on AFM-related queries, offering users flexibility in LLM selection based on their specific needs. LLM evaluations and the benchmark questions are available at: <uri>https://github.com/idealab-isu/AFM-LLM-Defect-Guidance</uri>.https://ieeexplore.ieee.org/document/11096088/atomic force microscopy (AFM)conversational LLMimage defect classificationlarge language models (LLMs) |
| spellingShingle | Angona Biswas Jaydeep Rade Nabila Masud Md Hasibul Hasan Hasib Aditya Balu Juntao Zhang Soumik Sarkar Adarsh Krishnamurthy Juan Ren Anwesha Sarkar Conversational LLM-Based Decision Support for Defect Classification in AFM Images IEEE Open Journal of Instrumentation and Measurement atomic force microscopy (AFM) conversational LLM image defect classification large language models (LLMs) |
| title | Conversational LLM-Based Decision Support for Defect Classification in AFM Images |
| title_full | Conversational LLM-Based Decision Support for Defect Classification in AFM Images |
| title_fullStr | Conversational LLM-Based Decision Support for Defect Classification in AFM Images |
| title_full_unstemmed | Conversational LLM-Based Decision Support for Defect Classification in AFM Images |
| title_short | Conversational LLM-Based Decision Support for Defect Classification in AFM Images |
| title_sort | conversational llm based decision support for defect classification in afm images |
| topic | atomic force microscopy (AFM) conversational LLM image defect classification large language models (LLMs) |
| url | https://ieeexplore.ieee.org/document/11096088/ |
| work_keys_str_mv | AT angonabiswas conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT jaydeeprade conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT nabilamasud conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT mdhasibulhasanhasib conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT adityabalu conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT juntaozhang conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT soumiksarkar conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT adarshkrishnamurthy conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT juanren conversationalllmbaseddecisionsupportfordefectclassificationinafmimages AT anweshasarkar conversationalllmbaseddecisionsupportfordefectclassificationinafmimages |