Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program
Abstract Background Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate t...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
SAGE Publishing
2023-08-01
|
| Series: | Journal of Otolaryngology - Head and Neck Surgery |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s40463-023-00649-2 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849683314905972736 |
|---|---|
| author | Mary Ott Tavis Apramian Sayra Cristancho Kathryn Roth |
| author_facet | Mary Ott Tavis Apramian Sayra Cristancho Kathryn Roth |
| author_sort | Mary Ott |
| collection | DOAJ |
| description | Abstract Background Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design. Methods We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program’s use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs. Results Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a “numbers game” residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning. Conclusion Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training. Graphical abstract |
| format | Article |
| id | doaj-art-dcb3004a419f4cd4a738ec2dae03eea9 |
| institution | DOAJ |
| issn | 1916-0216 |
| language | English |
| publishDate | 2023-08-01 |
| publisher | SAGE Publishing |
| record_format | Article |
| series | Journal of Otolaryngology - Head and Neck Surgery |
| spelling | doaj-art-dcb3004a419f4cd4a738ec2dae03eea92025-08-20T03:23:56ZengSAGE PublishingJournal of Otolaryngology - Head and Neck Surgery1916-02162023-08-015211810.1186/s40463-023-00649-2Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS programMary Ott0Tavis Apramian1Sayra Cristancho2Kathryn Roth3Centre for Education Research and Innovation, Schulich School of Medicine and Dentistry, Western UniversityDivision of Palliative Care, Department of Family & Community Medicine, University of TorontoCentre for Education Research and Innovation, Schulich School of Medicine and Dentistry, Western UniversityDepartment of Otolaryngology-Head and Neck Surgery, Schulich School of Medicine and Dentistry, Western UniversityAbstract Background Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design. Methods We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program’s use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs. Results Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a “numbers game” residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning. Conclusion Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training. Graphical abstracthttps://doi.org/10.1186/s40463-023-00649-2AssessmentCBMEe-portfolioTechnology designUnintended consequences |
| spellingShingle | Mary Ott Tavis Apramian Sayra Cristancho Kathryn Roth Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program Journal of Otolaryngology - Head and Neck Surgery Assessment CBME e-portfolio Technology design Unintended consequences |
| title | Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program |
| title_full | Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program |
| title_fullStr | Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program |
| title_full_unstemmed | Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program |
| title_short | Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program |
| title_sort | unintended consequences of technology in competency based education a qualitative study of lessons learned in an otohns program |
| topic | Assessment CBME e-portfolio Technology design Unintended consequences |
| url | https://doi.org/10.1186/s40463-023-00649-2 |
| work_keys_str_mv | AT maryott unintendedconsequencesoftechnologyincompetencybasededucationaqualitativestudyoflessonslearnedinanotohnsprogram AT tavisapramian unintendedconsequencesoftechnologyincompetencybasededucationaqualitativestudyoflessonslearnedinanotohnsprogram AT sayracristancho unintendedconsequencesoftechnologyincompetencybasededucationaqualitativestudyoflessonslearnedinanotohnsprogram AT kathrynroth unintendedconsequencesoftechnologyincompetencybasededucationaqualitativestudyoflessonslearnedinanotohnsprogram |