LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek
The integration of Large Language Models (LLMs) in chatbot applications gains momentum. However, to successfully deploy such systems, the underlying capabilities of LLMs must be carefully considered, especially when dealing with low-resource languages and specialized fields. This paper presents the...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-05-01
|
| Series: | Future Internet |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1999-5903/17/6/235 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849432886211510272 |
|---|---|
| author | Alexandros Tassios Stergios Tegos Christos Bouas Konstantinos Manousaridis Maria Papoutsoglou Maria Kaltsa Eleni Dimopoulou Thanassis Mavropoulos Stefanos Vrochidis Georgios Meditskos |
| author_facet | Alexandros Tassios Stergios Tegos Christos Bouas Konstantinos Manousaridis Maria Papoutsoglou Maria Kaltsa Eleni Dimopoulou Thanassis Mavropoulos Stefanos Vrochidis Georgios Meditskos |
| author_sort | Alexandros Tassios |
| collection | DOAJ |
| description | The integration of Large Language Models (LLMs) in chatbot applications gains momentum. However, to successfully deploy such systems, the underlying capabilities of LLMs must be carefully considered, especially when dealing with low-resource languages and specialized fields. This paper presents the results of a comprehensive evaluation of several LLMs conducted in the context of a chatbot agent designed to assist migrants in their integration process. Our aim is to identify the optimal LLM that can effectively process and generate text in Greek and provide accurate information, addressing the specific needs of migrant populations. The design of the evaluation methodology leverages input from experts on social assistance initiatives, social impact and technological solutions, as well as from automated LLM self-evaluations. Given the linguistic challenges specific to the Greek language and the application domain, research findings indicate that Claude 3.7 Sonnet and Gemini 2.0 Flash demonstrate superior performance across all criteria, with Claude 3.7 Sonnet emerging as the leading candidate for the chatbot. Moreover, the results suggest that automated custom evaluations of LLMs can align with human assessments, offering a viable option for preliminary low-cost analysis to assist stakeholders in selecting the optimal LLM based on user and application domain requirements. |
| format | Article |
| id | doaj-art-b6ab3fece33d4d7cb0a76d2b526d870c |
| institution | Kabale University |
| issn | 1999-5903 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Future Internet |
| spelling | doaj-art-b6ab3fece33d4d7cb0a76d2b526d870c2025-08-20T03:27:14ZengMDPI AGFuture Internet1999-59032025-05-0117623510.3390/fi17060235LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in GreekAlexandros Tassios0Stergios Tegos1Christos Bouas2Konstantinos Manousaridis3Maria Papoutsoglou4Maria Kaltsa5Eleni Dimopoulou6Thanassis Mavropoulos7Stefanos Vrochidis8Georgios Meditskos9School of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreeceSchool of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreeceSchool of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreeceSchool of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreeceSchool of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreeceDepartment of Theoretical & Applied Linguistics, School of English, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreecePRAKSIS, 10432 Athens, GreeceInformation Technologies Institute, Center for Research and Technology Hellas, 57001 Thessaloniki, GreeceInformation Technologies Institute, Center for Research and Technology Hellas, 57001 Thessaloniki, GreeceSchool of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, GreeceThe integration of Large Language Models (LLMs) in chatbot applications gains momentum. However, to successfully deploy such systems, the underlying capabilities of LLMs must be carefully considered, especially when dealing with low-resource languages and specialized fields. This paper presents the results of a comprehensive evaluation of several LLMs conducted in the context of a chatbot agent designed to assist migrants in their integration process. Our aim is to identify the optimal LLM that can effectively process and generate text in Greek and provide accurate information, addressing the specific needs of migrant populations. The design of the evaluation methodology leverages input from experts on social assistance initiatives, social impact and technological solutions, as well as from automated LLM self-evaluations. Given the linguistic challenges specific to the Greek language and the application domain, research findings indicate that Claude 3.7 Sonnet and Gemini 2.0 Flash demonstrate superior performance across all criteria, with Claude 3.7 Sonnet emerging as the leading candidate for the chatbot. Moreover, the results suggest that automated custom evaluations of LLMs can align with human assessments, offering a viable option for preliminary low-cost analysis to assist stakeholders in selecting the optimal LLM based on user and application domain requirements.https://www.mdpi.com/1999-5903/17/6/235chatbotGreek languagelarge language modelsLLM evaluationmigrant integration |
| spellingShingle | Alexandros Tassios Stergios Tegos Christos Bouas Konstantinos Manousaridis Maria Papoutsoglou Maria Kaltsa Eleni Dimopoulou Thanassis Mavropoulos Stefanos Vrochidis Georgios Meditskos LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek Future Internet chatbot Greek language large language models LLM evaluation migrant integration |
| title | LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek |
| title_full | LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek |
| title_fullStr | LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek |
| title_full_unstemmed | LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek |
| title_short | LLM Performance in Low-Resource Languages: Selecting an Optimal Model for Migrant Integration Support in Greek |
| title_sort | llm performance in low resource languages selecting an optimal model for migrant integration support in greek |
| topic | chatbot Greek language large language models LLM evaluation migrant integration |
| url | https://www.mdpi.com/1999-5903/17/6/235 |
| work_keys_str_mv | AT alexandrostassios llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT stergiostegos llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT christosbouas llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT konstantinosmanousaridis llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT mariapapoutsoglou llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT mariakaltsa llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT elenidimopoulou llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT thanassismavropoulos llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT stefanosvrochidis llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek AT georgiosmeditskos llmperformanceinlowresourcelanguagesselectinganoptimalmodelformigrantintegrationsupportingreek |