LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems
Ontologies are critical for organizing and interpreting complex domain-specific knowledge, with applications in data integration, functional prediction, and knowledge discovery. As the manual curation of ontology annotations becomes increasingly infeasible due to the exponential growth of biomedical...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-03-01
|
| Series: | Information |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2078-2489/16/3/225 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849341633368162304 |
|---|---|
| author | Ali Noori Pratik Devkota Somya D. Mohanty Prashanti Manda |
| author_facet | Ali Noori Pratik Devkota Somya D. Mohanty Prashanti Manda |
| author_sort | Ali Noori |
| collection | DOAJ |
| description | Ontologies are critical for organizing and interpreting complex domain-specific knowledge, with applications in data integration, functional prediction, and knowledge discovery. As the manual curation of ontology annotations becomes increasingly infeasible due to the exponential growth of biomedical and genomic data, natural language processing (NLP)-based systems have emerged as scalable alternatives. Evaluating these systems requires robust semantic similarity metrics that account for hierarchical and partially correct relationships often present in ontology annotations. This study explores the integration of graph-based and language-based embeddings to enhance the performance of semantic similarity metrics. Combining embeddings generated via Node2Vec and large language models (LLMs) with traditional semantic similarity metrics, we demonstrate that hybrid approaches effectively capture both structural and semantic relationships within ontologies. Our results show that combined similarity metrics outperform individual metrics, achieving high accuracy in distinguishing child–parent pairs from random pairs. This work underscores the importance of robust semantic similarity metrics for evaluating and optimizing NLP-based ontology annotation systems. Future research should explore the real-time integration of these metrics and advanced neural architectures to further enhance scalability and accuracy, advancing ontology-driven analyses in biomedical research and beyond. |
| format | Article |
| id | doaj-art-96da1f46ac1344ba8c4fb2d16a43dc35 |
| institution | Kabale University |
| issn | 2078-2489 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Information |
| spelling | doaj-art-96da1f46ac1344ba8c4fb2d16a43dc352025-08-20T03:43:36ZengMDPI AGInformation2078-24892025-03-0116322510.3390/info16030225LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation SystemsAli Noori0Pratik Devkota1Somya D. Mohanty2Prashanti Manda3Informatics and Analytics, University of North Carolina, Greensboro, NC 27412, USAFractal Analytics, New York, NY 10006, USAUnited Health Group, Minnetonka, MN 55343, USADepartment of Computer Science, University of Nebraska, Omaha, NE 68182, USAOntologies are critical for organizing and interpreting complex domain-specific knowledge, with applications in data integration, functional prediction, and knowledge discovery. As the manual curation of ontology annotations becomes increasingly infeasible due to the exponential growth of biomedical and genomic data, natural language processing (NLP)-based systems have emerged as scalable alternatives. Evaluating these systems requires robust semantic similarity metrics that account for hierarchical and partially correct relationships often present in ontology annotations. This study explores the integration of graph-based and language-based embeddings to enhance the performance of semantic similarity metrics. Combining embeddings generated via Node2Vec and large language models (LLMs) with traditional semantic similarity metrics, we demonstrate that hybrid approaches effectively capture both structural and semantic relationships within ontologies. Our results show that combined similarity metrics outperform individual metrics, achieving high accuracy in distinguishing child–parent pairs from random pairs. This work underscores the importance of robust semantic similarity metrics for evaluating and optimizing NLP-based ontology annotation systems. Future research should explore the real-time integration of these metrics and advanced neural architectures to further enhance scalability and accuracy, advancing ontology-driven analyses in biomedical research and beyond.https://www.mdpi.com/2078-2489/16/3/225semantic similarityLLMsontology annotationgene ontology |
| spellingShingle | Ali Noori Pratik Devkota Somya D. Mohanty Prashanti Manda LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems Information semantic similarity LLMs ontology annotation gene ontology |
| title | LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems |
| title_full | LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems |
| title_fullStr | LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems |
| title_full_unstemmed | LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems |
| title_short | LLMs in Action: Robust Metrics for Evaluating Automated Ontology Annotation Systems |
| title_sort | llms in action robust metrics for evaluating automated ontology annotation systems |
| topic | semantic similarity LLMs ontology annotation gene ontology |
| url | https://www.mdpi.com/2078-2489/16/3/225 |
| work_keys_str_mv | AT alinoori llmsinactionrobustmetricsforevaluatingautomatedontologyannotationsystems AT pratikdevkota llmsinactionrobustmetricsforevaluatingautomatedontologyannotationsystems AT somyadmohanty llmsinactionrobustmetricsforevaluatingautomatedontologyannotationsystems AT prashantimanda llmsinactionrobustmetricsforevaluatingautomatedontologyannotationsystems |