Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis.
Ankylosing spondylitis (AS), which usually occurs in the second and third decades of life, is associated with chronic pain, limitation of mobility, and severe decreases in quality of life. This study aimed to make a comparative evaluation in terms of the readability, information accuracy and quality...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2025-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0326351 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849329601758625792 |
|---|---|
| author | Mete Kara Erkan Ozduran Müge Mercan Kara İlhan Celil Özbek Volkan Hancı |
| author_facet | Mete Kara Erkan Ozduran Müge Mercan Kara İlhan Celil Özbek Volkan Hancı |
| author_sort | Mete Kara |
| collection | DOAJ |
| description | Ankylosing spondylitis (AS), which usually occurs in the second and third decades of life, is associated with chronic pain, limitation of mobility, and severe decreases in quality of life. This study aimed to make a comparative evaluation in terms of the readability, information accuracy and quality of the answers given by artificial intelligence (AI)-based chatbots such as ChatGPT, Perplexity and Gemini, which have become popular with the widespread access to medical information, to user questions about AS, a chronic inflammatory joint disease. In this study, the 25 most frequently queried keywords related to AS determined through Google Trends were directed to each 3 AI-based chatbots. The readability of the resulting responses was evaluated using readability indices such as Simple Gunning Fog (GFOG), Flesch Reading Ease Score (FRES) and Measure of Gobbledygook (SMOG). The quality of the responses was measured by Ensuring Quality Information for Patients (EQIP) and Global Quality Score (GQS) scores, and the reliability was measured using the modified DISCERN and Journal of American Medical Association (JAMA) scales. According to Google Trends data, the most frequently searched keywords related to AS are "Ankylosing spondylitis pain", "Ankylosing spondylitis symptoms" and "Ankylosing spondylitis disease", respectively. It was found that the readability levels of the answers produced by AI-based chatbots were above the 6th grade level and showed a statistically significant difference (p < 0.001). In EQIP, JAMA, mDISCERN and GQS evaluations, Perplexity stood out in terms of information quality and reliability, receiving higher scores compared to other chat robots (p < 0.05). It has been found that the answers given by AI chatbots to AS-related questions exceed the recommended readability level and the reliability and quality assessment raises concerns due to some low scores. It is possible for future AI chatbots to have sufficient quality, reliability and appropriate readability levels with an audit mechanism in place. |
| format | Article |
| id | doaj-art-21bd4c3849104d20860c64832f3dfd05 |
| institution | Kabale University |
| issn | 1932-6203 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-21bd4c3849104d20860c64832f3dfd052025-08-20T03:47:13ZengPublic Library of Science (PLoS)PLoS ONE1932-62032025-01-01206e032635110.1371/journal.pone.0326351Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis.Mete KaraErkan OzduranMüge Mercan Karaİlhan Celil ÖzbekVolkan HancıAnkylosing spondylitis (AS), which usually occurs in the second and third decades of life, is associated with chronic pain, limitation of mobility, and severe decreases in quality of life. This study aimed to make a comparative evaluation in terms of the readability, information accuracy and quality of the answers given by artificial intelligence (AI)-based chatbots such as ChatGPT, Perplexity and Gemini, which have become popular with the widespread access to medical information, to user questions about AS, a chronic inflammatory joint disease. In this study, the 25 most frequently queried keywords related to AS determined through Google Trends were directed to each 3 AI-based chatbots. The readability of the resulting responses was evaluated using readability indices such as Simple Gunning Fog (GFOG), Flesch Reading Ease Score (FRES) and Measure of Gobbledygook (SMOG). The quality of the responses was measured by Ensuring Quality Information for Patients (EQIP) and Global Quality Score (GQS) scores, and the reliability was measured using the modified DISCERN and Journal of American Medical Association (JAMA) scales. According to Google Trends data, the most frequently searched keywords related to AS are "Ankylosing spondylitis pain", "Ankylosing spondylitis symptoms" and "Ankylosing spondylitis disease", respectively. It was found that the readability levels of the answers produced by AI-based chatbots were above the 6th grade level and showed a statistically significant difference (p < 0.001). In EQIP, JAMA, mDISCERN and GQS evaluations, Perplexity stood out in terms of information quality and reliability, receiving higher scores compared to other chat robots (p < 0.05). It has been found that the answers given by AI chatbots to AS-related questions exceed the recommended readability level and the reliability and quality assessment raises concerns due to some low scores. It is possible for future AI chatbots to have sufficient quality, reliability and appropriate readability levels with an audit mechanism in place.https://doi.org/10.1371/journal.pone.0326351 |
| spellingShingle | Mete Kara Erkan Ozduran Müge Mercan Kara İlhan Celil Özbek Volkan Hancı Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. PLoS ONE |
| title | Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. |
| title_full | Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. |
| title_fullStr | Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. |
| title_full_unstemmed | Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. |
| title_short | Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about Ankylosing spondylitis. |
| title_sort | evaluating the readability quality and reliability of responses generated by chatgpt gemini and perplexity on the most commonly asked questions about ankylosing spondylitis |
| url | https://doi.org/10.1371/journal.pone.0326351 |
| work_keys_str_mv | AT metekara evaluatingthereadabilityqualityandreliabilityofresponsesgeneratedbychatgptgeminiandperplexityonthemostcommonlyaskedquestionsaboutankylosingspondylitis AT erkanozduran evaluatingthereadabilityqualityandreliabilityofresponsesgeneratedbychatgptgeminiandperplexityonthemostcommonlyaskedquestionsaboutankylosingspondylitis AT mugemercankara evaluatingthereadabilityqualityandreliabilityofresponsesgeneratedbychatgptgeminiandperplexityonthemostcommonlyaskedquestionsaboutankylosingspondylitis AT ilhancelilozbek evaluatingthereadabilityqualityandreliabilityofresponsesgeneratedbychatgptgeminiandperplexityonthemostcommonlyaskedquestionsaboutankylosingspondylitis AT volkanhancı evaluatingthereadabilityqualityandreliabilityofresponsesgeneratedbychatgptgeminiandperplexityonthemostcommonlyaskedquestionsaboutankylosingspondylitis |