Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI.
Recent research on Responsible AI, particularly in addressing algorithmic biases, has gained significant attention. Natural Language Processing (NLP) algorithms, which rely on human-generated and human-labeled data, often reflect these challenges. In this paper, we analyze inter-annotator agreement...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
LibraryPress@UF
2025-05-01
|
| Series: | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| Online Access: | https://journals.flvc.org/FLAIRS/article/view/139049 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850277108951023616 |
|---|---|
| author | Amir Toliyat Elena Filatova Ronak Etemadpour |
| author_facet | Amir Toliyat Elena Filatova Ronak Etemadpour |
| author_sort | Amir Toliyat |
| collection | DOAJ |
| description |
Recent research on Responsible AI, particularly in addressing algorithmic biases, has gained significant attention. Natural Language Processing (NLP) algorithms, which rely on human-generated and human-labeled data, often reflect these challenges. In this paper, we analyze inter-annotator agreement in the task of labeling hate speech data and examine how annotators’ backgrounds influence their labeling decisions. Specifically, we investigate differences in hate speech annotations that arise when annotators identify with the targeted groups. Our findings reveal substantial differences in agreement between a general pool of annotators and those who personally relate to the targets of the hate speech they label. Additionally, we evaluate the OpenAI GPT-4o model on the same dataset. Our results highlight the need to consider annotators’ backgrounds when assessing the performance of Large Language Models (LLMs) in hate speech detection.
|
| format | Article |
| id | doaj-art-7a97f4f78f7e4deb9ae1187546be9e23 |
| institution | OA Journals |
| issn | 2334-0754 2334-0762 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | LibraryPress@UF |
| record_format | Article |
| series | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| spelling | doaj-art-7a97f4f78f7e4deb9ae1187546be9e232025-08-20T01:49:59ZengLibraryPress@UFProceedings of the International Florida Artificial Intelligence Research Society Conference2334-07542334-07622025-05-0138110.32473/flairs.38.1.139049Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI.Amir Toliyat0Elena Filatova1Ronak EtemadpourVaughn College of Aeronautics and TechnologyCUNY Recent research on Responsible AI, particularly in addressing algorithmic biases, has gained significant attention. Natural Language Processing (NLP) algorithms, which rely on human-generated and human-labeled data, often reflect these challenges. In this paper, we analyze inter-annotator agreement in the task of labeling hate speech data and examine how annotators’ backgrounds influence their labeling decisions. Specifically, we investigate differences in hate speech annotations that arise when annotators identify with the targeted groups. Our findings reveal substantial differences in agreement between a general pool of annotators and those who personally relate to the targets of the hate speech they label. Additionally, we evaluate the OpenAI GPT-4o model on the same dataset. Our results highlight the need to consider annotators’ backgrounds when assessing the performance of Large Language Models (LLMs) in hate speech detection. https://journals.flvc.org/FLAIRS/article/view/139049 |
| spellingShingle | Amir Toliyat Elena Filatova Ronak Etemadpour Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI. Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| title | Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI. |
| title_full | Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI. |
| title_fullStr | Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI. |
| title_full_unstemmed | Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI. |
| title_short | Inter-Annotator Agreement and Its Reflection in LLMs and Responsible AI. |
| title_sort | inter annotator agreement and its reflection in llms and responsible ai |
| url | https://journals.flvc.org/FLAIRS/article/view/139049 |
| work_keys_str_mv | AT amirtoliyat interannotatoragreementanditsreflectioninllmsandresponsibleai AT elenafilatova interannotatoragreementanditsreflectioninllmsandresponsibleai AT ronaketemadpour interannotatoragreementanditsreflectioninllmsandresponsibleai |