Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach
IntroductionSituational judgment tests (SJT) are commonly used in admissions to measure skills associated with professionalism. Although open-response SJTs have shown strong psychometric properties, assessors’ personal beliefs, experiences, and cultural backgrounds may influence how they perceive, o...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-01-01
|
| Series: | Frontiers in Medicine |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/fmed.2024.1525156/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850078997107441664 |
|---|---|
| author | Muhammad Zafar Iqbal Rodica Ivan Colleen Robb Jillian Derby |
| author_facet | Muhammad Zafar Iqbal Rodica Ivan Colleen Robb Jillian Derby |
| author_sort | Muhammad Zafar Iqbal |
| collection | DOAJ |
| description | IntroductionSituational judgment tests (SJT) are commonly used in admissions to measure skills associated with professionalism. Although open-response SJTs have shown strong psychometric properties, assessors’ personal beliefs, experiences, and cultural backgrounds may influence how they perceive, organize and evaluate information within test takers’ diverse responses. Additionally, SJT research typically focuses on reliability and predictive validity, whereas the construct validity of open response SJTs remains underexplored. This mixed methods study aims to address this gap by exploring the construct-(ir)relevant factors that may impact assessors’ evaluation of professionalism in open response SJTs.MethodsFor this study, we used data from Casper, an open response SJT commonly used in professional program admissions. In Study I, a quantitative content analysis was conducted on 160 responses to identify factors which were significant predictors of low and high scores. Correlation coefficients and logistic regression models were used to evaluate the relationship between each factor and response scores. In Study II, think-aloud activities were conducted with 23 Casper assessors to directly observe how they evaluated responses. All interviews were transcribed verbatim, which were then thematically analyzed using an inductive coding technique.ResultsResults from both the content analyses and think-aloud activities revealed that several construct relevant factors influenced scores. Scores were impacted by the extent to which test takers demonstrated the competencies probed for by the SJT, engaged with the context of the presented ethical dilemma, provided in-depth justifications for their response, considered various perspectives relevant to the presented dilemma, and provided creative solutions or insightful arguments for the suggested approach. Mixed results were found with respect to construct irrelevant factors, such as the flow, cohesion, and kinds of phrases used in the response.ConclusionThis mixed methods study contributes to the construct validity of SJTs by investigating construct relevant and irrelevant factors that may impact assessors’ evaluation of open responses. The findings of this study provide evidence that open-response SJTs are valid approaches to measure professional competencies more broadly, both in terms of what test takers focus on in their responses, as well as in terms of how they construct their responses. |
| format | Article |
| id | doaj-art-889e5e9ee6264287a5aab7951b6c0aae |
| institution | DOAJ |
| issn | 2296-858X |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Medicine |
| spelling | doaj-art-889e5e9ee6264287a5aab7951b6c0aae2025-08-20T02:45:23ZengFrontiers Media S.A.Frontiers in Medicine2296-858X2025-01-011110.3389/fmed.2024.15251561525156Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approachMuhammad Zafar IqbalRodica IvanColleen RobbJillian DerbyIntroductionSituational judgment tests (SJT) are commonly used in admissions to measure skills associated with professionalism. Although open-response SJTs have shown strong psychometric properties, assessors’ personal beliefs, experiences, and cultural backgrounds may influence how they perceive, organize and evaluate information within test takers’ diverse responses. Additionally, SJT research typically focuses on reliability and predictive validity, whereas the construct validity of open response SJTs remains underexplored. This mixed methods study aims to address this gap by exploring the construct-(ir)relevant factors that may impact assessors’ evaluation of professionalism in open response SJTs.MethodsFor this study, we used data from Casper, an open response SJT commonly used in professional program admissions. In Study I, a quantitative content analysis was conducted on 160 responses to identify factors which were significant predictors of low and high scores. Correlation coefficients and logistic regression models were used to evaluate the relationship between each factor and response scores. In Study II, think-aloud activities were conducted with 23 Casper assessors to directly observe how they evaluated responses. All interviews were transcribed verbatim, which were then thematically analyzed using an inductive coding technique.ResultsResults from both the content analyses and think-aloud activities revealed that several construct relevant factors influenced scores. Scores were impacted by the extent to which test takers demonstrated the competencies probed for by the SJT, engaged with the context of the presented ethical dilemma, provided in-depth justifications for their response, considered various perspectives relevant to the presented dilemma, and provided creative solutions or insightful arguments for the suggested approach. Mixed results were found with respect to construct irrelevant factors, such as the flow, cohesion, and kinds of phrases used in the response.ConclusionThis mixed methods study contributes to the construct validity of SJTs by investigating construct relevant and irrelevant factors that may impact assessors’ evaluation of open responses. The findings of this study provide evidence that open-response SJTs are valid approaches to measure professional competencies more broadly, both in terms of what test takers focus on in their responses, as well as in terms of how they construct their responses.https://www.frontiersin.org/articles/10.3389/fmed.2024.1525156/fullsituational judgment testsopen response scoringconstruct validityprofessionalismpersonal skillsprofessional skills |
| spellingShingle | Muhammad Zafar Iqbal Rodica Ivan Colleen Robb Jillian Derby Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach Frontiers in Medicine situational judgment tests open response scoring construct validity professionalism personal skills professional skills |
| title | Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach |
| title_full | Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach |
| title_fullStr | Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach |
| title_full_unstemmed | Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach |
| title_short | Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach |
| title_sort | evaluating factors that impact scoring an open response situational judgment test a mixed methods approach |
| topic | situational judgment tests open response scoring construct validity professionalism personal skills professional skills |
| url | https://www.frontiersin.org/articles/10.3389/fmed.2024.1525156/full |
| work_keys_str_mv | AT muhammadzafariqbal evaluatingfactorsthatimpactscoringanopenresponsesituationaljudgmenttestamixedmethodsapproach AT rodicaivan evaluatingfactorsthatimpactscoringanopenresponsesituationaljudgmenttestamixedmethodsapproach AT colleenrobb evaluatingfactorsthatimpactscoringanopenresponsesituationaljudgmenttestamixedmethodsapproach AT jillianderby evaluatingfactorsthatimpactscoringanopenresponsesituationaljudgmenttestamixedmethodsapproach |