Assessment of Retrieval and Generative Chatbots in Tourism Information Service
Chatbots are essential for improving the customer experience on tourism websites, especially when it comes to arranging travel and offering precise information. The purpose of this study is to evaluate the effectiveness of generative and retrieval-based chatbots in the tourism information service. T...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | Indonesian |
| Published: |
Universitas Muhammadiyah Purwokerto
2025-03-01
|
| Series: | Jurnal Informatika |
| Subjects: | |
| Online Access: | http://jurnalnasional.ump.ac.id/index.php/JUITA/article/view/24182 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849243618261336064 |
|---|---|
| author | Sharfina Febbi Handayani Dairoh Dairoh Dwi Intan Af'idah |
| author_facet | Sharfina Febbi Handayani Dairoh Dairoh Dwi Intan Af'idah |
| author_sort | Sharfina Febbi Handayani |
| collection | DOAJ |
| description | Chatbots are essential for improving the customer experience on tourism websites, especially when it comes to arranging travel and offering precise information. The purpose of this study is to evaluate the effectiveness of generative and retrieval-based chatbots in the tourism information service. Two retrieval-based models are MLP-based single QA and multi QA and two generative-based models namely LLaMA 2 and GEMMA were evaluated using confusion matrix, BLUE score, correctness response and naturalness response. The study found that LLaMA 2 outperformed other models, with the highest response Accuracy of 0.89, naturalness of 0.75, and BLEU score of 0.33. GEMMA received the lowest score, suggesting that it has trouble coming up with precise and organic answers. The retrieval-based models showed strong accuracy but were less natural in their responses. The ease of dataset creation for generative models, which only requires narrative text, further positions LLaMA 2 as the most suitable option for improving user experience in Tegal tourism services. |
| format | Article |
| id | doaj-art-a9c3f45173b4452d8bd482bb1d83f2e5 |
| institution | Kabale University |
| issn | 2086-9398 2579-8901 |
| language | Indonesian |
| publishDate | 2025-03-01 |
| publisher | Universitas Muhammadiyah Purwokerto |
| record_format | Article |
| series | Jurnal Informatika |
| spelling | doaj-art-a9c3f45173b4452d8bd482bb1d83f2e52025-08-20T03:59:25ZindUniversitas Muhammadiyah PurwokertoJurnal Informatika2086-93982579-89012025-03-01192710.30595/juita.v13i1.2418219526Assessment of Retrieval and Generative Chatbots in Tourism Information ServiceSharfina Febbi Handayani0Dairoh Dairoh1Dwi Intan Af'idah2Politeknik Harapan BersamaPoliteknik Harapan BersamaPoliteknik Harapan BersamaChatbots are essential for improving the customer experience on tourism websites, especially when it comes to arranging travel and offering precise information. The purpose of this study is to evaluate the effectiveness of generative and retrieval-based chatbots in the tourism information service. Two retrieval-based models are MLP-based single QA and multi QA and two generative-based models namely LLaMA 2 and GEMMA were evaluated using confusion matrix, BLUE score, correctness response and naturalness response. The study found that LLaMA 2 outperformed other models, with the highest response Accuracy of 0.89, naturalness of 0.75, and BLEU score of 0.33. GEMMA received the lowest score, suggesting that it has trouble coming up with precise and organic answers. The retrieval-based models showed strong accuracy but were less natural in their responses. The ease of dataset creation for generative models, which only requires narrative text, further positions LLaMA 2 as the most suitable option for improving user experience in Tegal tourism services.http://jurnalnasional.ump.ac.id/index.php/JUITA/article/view/24182chatbotgemmatllams2tourism. |
| spellingShingle | Sharfina Febbi Handayani Dairoh Dairoh Dwi Intan Af'idah Assessment of Retrieval and Generative Chatbots in Tourism Information Service Jurnal Informatika chatbot gemma tllams2 tourism. |
| title | Assessment of Retrieval and Generative Chatbots in Tourism Information Service |
| title_full | Assessment of Retrieval and Generative Chatbots in Tourism Information Service |
| title_fullStr | Assessment of Retrieval and Generative Chatbots in Tourism Information Service |
| title_full_unstemmed | Assessment of Retrieval and Generative Chatbots in Tourism Information Service |
| title_short | Assessment of Retrieval and Generative Chatbots in Tourism Information Service |
| title_sort | assessment of retrieval and generative chatbots in tourism information service |
| topic | chatbot gemma tllams2 tourism. |
| url | http://jurnalnasional.ump.ac.id/index.php/JUITA/article/view/24182 |
| work_keys_str_mv | AT sharfinafebbihandayani assessmentofretrievalandgenerativechatbotsintourisminformationservice AT dairohdairoh assessmentofretrievalandgenerativechatbotsintourisminformationservice AT dwiintanafidah assessmentofretrievalandgenerativechatbotsintourisminformationservice |