Taking disagreements into consideration: human annotation variability in privacy policy analysis
Introduction. Privacy policies inform users about data practices but are often complex and difficult to interpret. Human annotation plays a key role in understanding privacy policies, yet annotation disagreements highlight the complexity of these texts. Traditional machine learning models prioritiz...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
University of Borås
2025-03-01
|
| Series: | Information Research: An International Electronic Journal |
| Subjects: | |
| Online Access: | https://publicera.kb.se/ir/article/view/47581 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850038032534601728 |
|---|---|
| author | Tian Wang Yuanye Ma Catherine Blake Masooda Bashir Hsin-Yuan Wang |
| author_facet | Tian Wang Yuanye Ma Catherine Blake Masooda Bashir Hsin-Yuan Wang |
| author_sort | Tian Wang |
| collection | DOAJ |
| description |
Introduction. Privacy policies inform users about data practices but are often complex and difficult to interpret. Human annotation plays a key role in understanding privacy policies, yet annotation disagreements highlight the complexity of these texts. Traditional machine learning models prioritize consensus, overlooking annotation variability and its impact on accuracy.
Method. This study examines how annotation disagreements affect machine learning performance using the OPP-115 corpus. It compares majority vote and union methods with alternative strategies to assess their impact on policy classification.
Analysis. The study evaluates whether increasing annotator consensus improves model effectiveness and if disagreement-aware approaches yield more reliable results.
Results. Higher agreement levels improve model performance across most categories. Complete agreement yields the best F1-scores, especially for First Party Collection/Use and Third-Party Sharing/Collection. Annotation disagreements significantly impact classification outcomes, underscoring the need for understanding annotation disagreements.
Conclusions. Ignoring annotation disagreements can misrepresent model accuracy. This study proposes new evaluation strategies that account for annotation variability, offering a more realistic approach to privacy policy analysis. Future work should explore the causes of annotation disagreements to improve machine learning transparency and reliability.
|
| format | Article |
| id | doaj-art-e85b146489b84dbfa5c0f1b8f1ef5113 |
| institution | DOAJ |
| issn | 1368-1613 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | University of Borås |
| record_format | Article |
| series | Information Research: An International Electronic Journal |
| spelling | doaj-art-e85b146489b84dbfa5c0f1b8f1ef51132025-08-20T02:56:43ZengUniversity of BoråsInformation Research: An International Electronic Journal1368-16132025-03-0130iConf10.47989/ir30iConf47581Taking disagreements into consideration: human annotation variability in privacy policy analysisTian Wang0Yuanye Ma1Catherine Blake2Masooda Bashir3Hsin-Yuan Wang4University of Illinois Urbana-ChampaignUniversity of Illinois Discovery Partners InstituteUniversity of Illinois Urbana-ChampaignUniversity of Illinois Urbana-ChampaignUniversity of Illinois Urbana-Champaign Introduction. Privacy policies inform users about data practices but are often complex and difficult to interpret. Human annotation plays a key role in understanding privacy policies, yet annotation disagreements highlight the complexity of these texts. Traditional machine learning models prioritize consensus, overlooking annotation variability and its impact on accuracy. Method. This study examines how annotation disagreements affect machine learning performance using the OPP-115 corpus. It compares majority vote and union methods with alternative strategies to assess their impact on policy classification. Analysis. The study evaluates whether increasing annotator consensus improves model effectiveness and if disagreement-aware approaches yield more reliable results. Results. Higher agreement levels improve model performance across most categories. Complete agreement yields the best F1-scores, especially for First Party Collection/Use and Third-Party Sharing/Collection. Annotation disagreements significantly impact classification outcomes, underscoring the need for understanding annotation disagreements. Conclusions. Ignoring annotation disagreements can misrepresent model accuracy. This study proposes new evaluation strategies that account for annotation variability, offering a more realistic approach to privacy policy analysis. Future work should explore the causes of annotation disagreements to improve machine learning transparency and reliability. https://publicera.kb.se/ir/article/view/47581privacy policyannotator disagreementnatural language processingmachine learninghuman label variation |
| spellingShingle | Tian Wang Yuanye Ma Catherine Blake Masooda Bashir Hsin-Yuan Wang Taking disagreements into consideration: human annotation variability in privacy policy analysis Information Research: An International Electronic Journal privacy policy annotator disagreement natural language processing machine learning human label variation |
| title | Taking disagreements into consideration: human annotation variability in privacy policy analysis |
| title_full | Taking disagreements into consideration: human annotation variability in privacy policy analysis |
| title_fullStr | Taking disagreements into consideration: human annotation variability in privacy policy analysis |
| title_full_unstemmed | Taking disagreements into consideration: human annotation variability in privacy policy analysis |
| title_short | Taking disagreements into consideration: human annotation variability in privacy policy analysis |
| title_sort | taking disagreements into consideration human annotation variability in privacy policy analysis |
| topic | privacy policy annotator disagreement natural language processing machine learning human label variation |
| url | https://publicera.kb.se/ir/article/view/47581 |
| work_keys_str_mv | AT tianwang takingdisagreementsintoconsiderationhumanannotationvariabilityinprivacypolicyanalysis AT yuanyema takingdisagreementsintoconsiderationhumanannotationvariabilityinprivacypolicyanalysis AT catherineblake takingdisagreementsintoconsiderationhumanannotationvariabilityinprivacypolicyanalysis AT masoodabashir takingdisagreementsintoconsiderationhumanannotationvariabilityinprivacypolicyanalysis AT hsinyuanwang takingdisagreementsintoconsiderationhumanannotationvariabilityinprivacypolicyanalysis |