Towards Automated Error Analysis: Learning to Characterize Errors
Characterizing the patterns of errors that a system makes helps researchers focus future development on increasing its accuracy and robustness. We propose a novel form of "meta learning" that automatically learns interpretable rules that characterize the types of errors that a system makes...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
LibraryPress@UF
2022-05-01
|
| Series: | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| Online Access: | https://journals.flvc.org/FLAIRS/article/view/130632 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849763276589629440 |
|---|---|
| author | Tong Gao Shivang Singh Raymond Mooney |
| author_facet | Tong Gao Shivang Singh Raymond Mooney |
| author_sort | Tong Gao |
| collection | DOAJ |
| description | Characterizing the patterns of errors that a system makes helps researchers focus future development on increasing its accuracy and robustness. We propose a novel form of "meta learning" that automatically learns interpretable rules that characterize the types of errors that a system makes, and demonstrate these rules' ability to help understand and improve two NLP systems. Our approach works by collecting error cases on validation data, extracting meta-features describing these samples, and finally learning rules that characterize errors using these features. We apply our approach to VilBERT, for Visual Question Answering, and RoBERTa, for Common Sense Question Answering. Our system learns interpretable rules that provide insights into systemic errors these systems make on the given tasks. Using these insights, we are also able to `"close the loop" and modestly improve performance of these systems. |
| format | Article |
| id | doaj-art-de8bf9c6ee894bfbb0ea8596ad39a0b9 |
| institution | DOAJ |
| issn | 2334-0754 2334-0762 |
| language | English |
| publishDate | 2022-05-01 |
| publisher | LibraryPress@UF |
| record_format | Article |
| series | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| spelling | doaj-art-de8bf9c6ee894bfbb0ea8596ad39a0b92025-08-20T03:05:26ZengLibraryPress@UFProceedings of the International Florida Artificial Intelligence Research Society Conference2334-07542334-07622022-05-013510.32473/flairs.v35i.13063266831Towards Automated Error Analysis: Learning to Characterize ErrorsTong Gao0Shivang SinghRaymond MooneyUniversity of Texas at AustinCharacterizing the patterns of errors that a system makes helps researchers focus future development on increasing its accuracy and robustness. We propose a novel form of "meta learning" that automatically learns interpretable rules that characterize the types of errors that a system makes, and demonstrate these rules' ability to help understand and improve two NLP systems. Our approach works by collecting error cases on validation data, extracting meta-features describing these samples, and finally learning rules that characterize errors using these features. We apply our approach to VilBERT, for Visual Question Answering, and RoBERTa, for Common Sense Question Answering. Our system learns interpretable rules that provide insights into systemic errors these systems make on the given tasks. Using these insights, we are also able to `"close the loop" and modestly improve performance of these systems.https://journals.flvc.org/FLAIRS/article/view/130632 |
| spellingShingle | Tong Gao Shivang Singh Raymond Mooney Towards Automated Error Analysis: Learning to Characterize Errors Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| title | Towards Automated Error Analysis: Learning to Characterize Errors |
| title_full | Towards Automated Error Analysis: Learning to Characterize Errors |
| title_fullStr | Towards Automated Error Analysis: Learning to Characterize Errors |
| title_full_unstemmed | Towards Automated Error Analysis: Learning to Characterize Errors |
| title_short | Towards Automated Error Analysis: Learning to Characterize Errors |
| title_sort | towards automated error analysis learning to characterize errors |
| url | https://journals.flvc.org/FLAIRS/article/view/130632 |
| work_keys_str_mv | AT tonggao towardsautomatederroranalysislearningtocharacterizeerrors AT shivangsingh towardsautomatederroranalysislearningtocharacterizeerrors AT raymondmooney towardsautomatederroranalysislearningtocharacterizeerrors |