Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology
Background: While chatbots such as Chat Generative Pre-Trained Transformer (ChatGPT) and BARD have gained significant popularity in aiding researchers in crafting scientific articles, concerns persist, particularly regarding their accuracy in generating references. This is especially pronounced in s...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-03-01
|
| Series: | Oral Oncology Reports |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2772906024000141 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841553780446855168 |
|---|---|
| author | Gargi Sarode Shruti Singh Sachin C. Sarode |
| author_facet | Gargi Sarode Shruti Singh Sachin C. Sarode |
| author_sort | Gargi Sarode |
| collection | DOAJ |
| description | Background: While chatbots such as Chat Generative Pre-Trained Transformer (ChatGPT) and BARD have gained significant popularity in aiding researchers in crafting scientific articles, concerns persist, particularly regarding their accuracy in generating references. This is especially pronounced in specialized fields like oral pathology, where precision is paramount. Methods: We conducted a comprehensive comparison between ChatGPT and BARD in the context of generating references for five frequently researched topics within oral pathology. To assess accuracy, we subjected fifty references generated by each model to rigorous evaluation based on multiple parameters. The validation process involved meticulous cross-referencing across prominent scientific databases. Results: ChatGPT demonstrated a reference accuracy of only 10 %, revealing notable inconsistencies across various parameters. Conversely, BARD exhibited a substantial 88 % inaccuracy rate in total references, with only 4 references managing to provide the correct title. Notably, the DOI led to the accurate manuscript in only 12 % (9) of references generated by ChatGPT and 4 % (8) of references from BARD. Analysis indicated significantly better performance by the ChatGPT in comparison to BARD (p < 0.05). Conclusion: Prudence is advised when incorporating references generated by ChatGPT and BARD on topics related to oral pathology. There is a pressing need for publishers to fortify systems for scrutinizing references and integrate reputable scientific databases into Chatbot systems to ensure accuracy. |
| format | Article |
| id | doaj-art-e0be5acf00cd41089e6ed2272b1cd7c2 |
| institution | Kabale University |
| issn | 2772-9060 |
| language | English |
| publishDate | 2024-03-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Oral Oncology Reports |
| spelling | doaj-art-e0be5acf00cd41089e6ed2272b1cd7c22025-01-09T06:15:12ZengElsevierOral Oncology Reports2772-90602024-03-019100168Cautionary note: Chatbots generate fictitious references on the topics related to oral pathologyGargi Sarode0Shruti Singh1Sachin C. Sarode2Department of Oral Pathology and Microbiology, Dr. D.Y. Patil Dental College and Hospital, Dr.D. Y. Patil Vidyapeeth, Pimpri, Pune, 411018, IndiaDepartment of Oral Pathology and Microbiology, Dr. D.Y. Patil Dental College and Hospital, Dr.D. Y. Patil Vidyapeeth, Pimpri, Pune, 411018, IndiaDepartment of Oral Pathology and Microbiology, Dr. D.Y. Patil Dental College and Hospital, Dr.D. Y. Patil Vidyapeeth, Pimpri, Pune, 411018, India; Dr. D. Y. Patil Unitech Society's, Dr. D. Y. Patil Institute of Pharmaceutical Sciences and Research, Pimpri Pune, 411018, Maharashtra, India; Corresponding author. Department of Oral Pathology and Microbiology, Dr. D.Y. Patil Dental College and Hospital, Dr. D.Y. Patil Vidyapeeth, Sant-Tukaram Nagar, Pimpri, Pune, 18, Maharashtra, India.Background: While chatbots such as Chat Generative Pre-Trained Transformer (ChatGPT) and BARD have gained significant popularity in aiding researchers in crafting scientific articles, concerns persist, particularly regarding their accuracy in generating references. This is especially pronounced in specialized fields like oral pathology, where precision is paramount. Methods: We conducted a comprehensive comparison between ChatGPT and BARD in the context of generating references for five frequently researched topics within oral pathology. To assess accuracy, we subjected fifty references generated by each model to rigorous evaluation based on multiple parameters. The validation process involved meticulous cross-referencing across prominent scientific databases. Results: ChatGPT demonstrated a reference accuracy of only 10 %, revealing notable inconsistencies across various parameters. Conversely, BARD exhibited a substantial 88 % inaccuracy rate in total references, with only 4 references managing to provide the correct title. Notably, the DOI led to the accurate manuscript in only 12 % (9) of references generated by ChatGPT and 4 % (8) of references from BARD. Analysis indicated significantly better performance by the ChatGPT in comparison to BARD (p < 0.05). Conclusion: Prudence is advised when incorporating references generated by ChatGPT and BARD on topics related to oral pathology. There is a pressing need for publishers to fortify systems for scrutinizing references and integrate reputable scientific databases into Chatbot systems to ensure accuracy.http://www.sciencedirect.com/science/article/pii/S2772906024000141AI-Generated referencesOral pathologyChatbot accuracyScientific citationReference validation |
| spellingShingle | Gargi Sarode Shruti Singh Sachin C. Sarode Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology Oral Oncology Reports AI-Generated references Oral pathology Chatbot accuracy Scientific citation Reference validation |
| title | Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology |
| title_full | Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology |
| title_fullStr | Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology |
| title_full_unstemmed | Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology |
| title_short | Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology |
| title_sort | cautionary note chatbots generate fictitious references on the topics related to oral pathology |
| topic | AI-Generated references Oral pathology Chatbot accuracy Scientific citation Reference validation |
| url | http://www.sciencedirect.com/science/article/pii/S2772906024000141 |
| work_keys_str_mv | AT gargisarode cautionarynotechatbotsgeneratefictitiousreferencesonthetopicsrelatedtooralpathology AT shrutisingh cautionarynotechatbotsgeneratefictitiousreferencesonthetopicsrelatedtooralpathology AT sachincsarode cautionarynotechatbotsgeneratefictitiousreferencesonthetopicsrelatedtooralpathology |