Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study
BackgroundIncreasing use of computational methods in health care provides opportunities to address previously unsolvable problems. Machine learning techniques applied to routinely collected data can enhance clinical tools and improve patient outcomes, but their effective depl...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
JMIR Publications
2025-02-01
|
| Series: | JMIR Human Factors |
| Online Access: | https://humanfactors.jmir.org/2025/1/e56880 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832088138732797952 |
|---|---|
| author | Marceli Wac Raul Santos-Rodriguez Chris McWilliams Christopher Bourdeaux |
| author_facet | Marceli Wac Raul Santos-Rodriguez Chris McWilliams Christopher Bourdeaux |
| author_sort | Marceli Wac |
| collection | DOAJ |
| description |
BackgroundIncreasing use of computational methods in health care provides opportunities to address previously unsolvable problems. Machine learning techniques applied to routinely collected data can enhance clinical tools and improve patient outcomes, but their effective deployment comes with significant challenges. While some tasks can be addressed by training machine learning models directly on the collected data, more complex problems require additional input in the form of data annotations. Data annotation is a complex and time-consuming problem that requires domain expertise and frequently, technical proficiency. With clinicians’ time being an extremely limited resource, existing tools fail to provide an effective workflow for deployment in health care.
ObjectiveThis paper investigates the approach of intensive care unit staff to the task of data annotation. Specifically, it aims to (1) understand how clinicians approach data annotation and (2) capture the requirements for a digital annotation tool for the health care setting.
MethodsWe conducted an experimental activity involving annotation of the printed excerpts of real time-series admission data with 7 intensive care unit clinicians. Each participant annotated an identical set of admissions with the periods of weaning from mechanical ventilation during a single 45-minute workshop. Participants were observed during task completion and their actions were analyzed within Norman’s Interaction Cycle model to identify the software requirements.
ResultsClinicians followed a cyclic process of investigation, annotation, data reevaluation, and label refinement. Variety of techniques were used to investigate data and create annotations. We identified 11 requirements for the digital tool across 4 domains: annotation of individual admissions (n=5), semiautomated annotation (n=3), operational constraints (n=2), and use of labels in machine learning (n=1).
ConclusionsEffective data annotation in a clinical setting relies on flexibility in analysis and label creation and workflow continuity across multiple admissions. There is a need to ensure a seamless transition between data investigation, annotation, and refinement of the labels. |
| format | Article |
| id | doaj-art-00f4a1a18e9b4b68beb6c40d7fbe5d06 |
| institution | Kabale University |
| issn | 2292-9495 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | JMIR Publications |
| record_format | Article |
| series | JMIR Human Factors |
| spelling | doaj-art-00f4a1a18e9b4b68beb6c40d7fbe5d062025-02-05T20:30:58ZengJMIR PublicationsJMIR Human Factors2292-94952025-02-0112e5688010.2196/56880Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design StudyMarceli Wachttps://orcid.org/0000-0002-1986-0401Raul Santos-Rodriguezhttps://orcid.org/0000-0001-9576-3905Chris McWilliamshttps://orcid.org/0000-0003-3816-5217Christopher Bourdeauxhttps://orcid.org/0000-0001-6620-6536 BackgroundIncreasing use of computational methods in health care provides opportunities to address previously unsolvable problems. Machine learning techniques applied to routinely collected data can enhance clinical tools and improve patient outcomes, but their effective deployment comes with significant challenges. While some tasks can be addressed by training machine learning models directly on the collected data, more complex problems require additional input in the form of data annotations. Data annotation is a complex and time-consuming problem that requires domain expertise and frequently, technical proficiency. With clinicians’ time being an extremely limited resource, existing tools fail to provide an effective workflow for deployment in health care. ObjectiveThis paper investigates the approach of intensive care unit staff to the task of data annotation. Specifically, it aims to (1) understand how clinicians approach data annotation and (2) capture the requirements for a digital annotation tool for the health care setting. MethodsWe conducted an experimental activity involving annotation of the printed excerpts of real time-series admission data with 7 intensive care unit clinicians. Each participant annotated an identical set of admissions with the periods of weaning from mechanical ventilation during a single 45-minute workshop. Participants were observed during task completion and their actions were analyzed within Norman’s Interaction Cycle model to identify the software requirements. ResultsClinicians followed a cyclic process of investigation, annotation, data reevaluation, and label refinement. Variety of techniques were used to investigate data and create annotations. We identified 11 requirements for the digital tool across 4 domains: annotation of individual admissions (n=5), semiautomated annotation (n=3), operational constraints (n=2), and use of labels in machine learning (n=1). ConclusionsEffective data annotation in a clinical setting relies on flexibility in analysis and label creation and workflow continuity across multiple admissions. There is a need to ensure a seamless transition between data investigation, annotation, and refinement of the labels.https://humanfactors.jmir.org/2025/1/e56880 |
| spellingShingle | Marceli Wac Raul Santos-Rodriguez Chris McWilliams Christopher Bourdeaux Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study JMIR Human Factors |
| title | Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study |
| title_full | Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study |
| title_fullStr | Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study |
| title_full_unstemmed | Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study |
| title_short | Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study |
| title_sort | capturing requirements for a data annotation tool for intensive care experimental user centered design study |
| url | https://humanfactors.jmir.org/2025/1/e56880 |
| work_keys_str_mv | AT marceliwac capturingrequirementsforadataannotationtoolforintensivecareexperimentalusercentereddesignstudy AT raulsantosrodriguez capturingrequirementsforadataannotationtoolforintensivecareexperimentalusercentereddesignstudy AT chrismcwilliams capturingrequirementsforadataannotationtoolforintensivecareexperimentalusercentereddesignstudy AT christopherbourdeaux capturingrequirementsforadataannotationtoolforintensivecareexperimentalusercentereddesignstudy |