Using Generalizability Theory to examine the effects of presentation order on classroom observation scores
BackgroundScores from classroom observations serve as important pieces of information for educational researchers, practitioners, and policymakers. However, scoring procedures typically remain untested. This study aims to investigate the effect of sequential and nonsequential scoring of lesson segme...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-05-01
|
| Series: | Frontiers in Education |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/feduc.2025.1547204/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850190224580149248 |
|---|---|
| author | Armin Jentsch Bas Senden Nani Teig Trude Nilsen Johannes König Gabriele Kaiser Gabriele Kaiser |
| author_facet | Armin Jentsch Bas Senden Nani Teig Trude Nilsen Johannes König Gabriele Kaiser Gabriele Kaiser |
| author_sort | Armin Jentsch |
| collection | DOAJ |
| description | BackgroundScores from classroom observations serve as important pieces of information for educational researchers, practitioners, and policymakers. However, scoring procedures typically remain untested. This study aims to investigate the effect of sequential and nonsequential scoring of lesson segments from videotaped Norwegian mathematics and science classrooms.MethodsUsing a cross-over design, we randomly assigned four raters to score 30 videotaped lessons in sequential (SEQ) or nonsequential (NON) 20-min segments. We explored means as well as correlations across conditions and conducted Generalizability Studies (G Studies) to estimate the precision of scores.ResultsWe found marginal fluctuations in scores across conditions, and correlation analysis indicated similar rankings of lessons and classrooms. However, G Studies reveal that nonsequential scoring captures a greater proportion of between-classroom variance, while sequential scoring captures more variance between lessons. This also implies that the precision of scores varies across conditions and the unit of analysis.ConclusionThe findings suggest that scoring procedures could affect decision-making drawing on scores from classroom observation. We argue, however, that results depend to some extent on the observation system used to generate the scores. Therefore, we encourage researchers to replicate our findings with other observation systems. |
| format | Article |
| id | doaj-art-528eba6cd9294a7fb5403adb6850d5df |
| institution | OA Journals |
| issn | 2504-284X |
| language | English |
| publishDate | 2025-05-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Education |
| spelling | doaj-art-528eba6cd9294a7fb5403adb6850d5df2025-08-20T02:15:20ZengFrontiers Media S.A.Frontiers in Education2504-284X2025-05-011010.3389/feduc.2025.15472041547204Using Generalizability Theory to examine the effects of presentation order on classroom observation scoresArmin Jentsch0Bas Senden1Nani Teig2Trude Nilsen3Johannes König4Gabriele Kaiser5Gabriele Kaiser6Department of Teacher Education and School Research, University of Oslo, Oslo, NorwayDepartment of Teacher Education and School Research, University of Oslo, Oslo, NorwayDepartment of Teacher Education and School Research, University of Oslo, Oslo, NorwayDepartment of Teacher Education and School Research, University of Oslo, Oslo, NorwayFaculty of Human Sciences, University of Cologne, Cologne, GermanyFaculty of Education, University of Hamburg, Hamburg, GermanyFaculty of Education and Arts, Nord University, Bodø, NorwayBackgroundScores from classroom observations serve as important pieces of information for educational researchers, practitioners, and policymakers. However, scoring procedures typically remain untested. This study aims to investigate the effect of sequential and nonsequential scoring of lesson segments from videotaped Norwegian mathematics and science classrooms.MethodsUsing a cross-over design, we randomly assigned four raters to score 30 videotaped lessons in sequential (SEQ) or nonsequential (NON) 20-min segments. We explored means as well as correlations across conditions and conducted Generalizability Studies (G Studies) to estimate the precision of scores.ResultsWe found marginal fluctuations in scores across conditions, and correlation analysis indicated similar rankings of lessons and classrooms. However, G Studies reveal that nonsequential scoring captures a greater proportion of between-classroom variance, while sequential scoring captures more variance between lessons. This also implies that the precision of scores varies across conditions and the unit of analysis.ConclusionThe findings suggest that scoring procedures could affect decision-making drawing on scores from classroom observation. We argue, however, that results depend to some extent on the observation system used to generate the scores. Therefore, we encourage researchers to replicate our findings with other observation systems.https://www.frontiersin.org/articles/10.3389/feduc.2025.1547204/fullteaching qualityobservation systemsgeneralizability theoryclassroom observationvalidity |
| spellingShingle | Armin Jentsch Bas Senden Nani Teig Trude Nilsen Johannes König Gabriele Kaiser Gabriele Kaiser Using Generalizability Theory to examine the effects of presentation order on classroom observation scores Frontiers in Education teaching quality observation systems generalizability theory classroom observation validity |
| title | Using Generalizability Theory to examine the effects of presentation order on classroom observation scores |
| title_full | Using Generalizability Theory to examine the effects of presentation order on classroom observation scores |
| title_fullStr | Using Generalizability Theory to examine the effects of presentation order on classroom observation scores |
| title_full_unstemmed | Using Generalizability Theory to examine the effects of presentation order on classroom observation scores |
| title_short | Using Generalizability Theory to examine the effects of presentation order on classroom observation scores |
| title_sort | using generalizability theory to examine the effects of presentation order on classroom observation scores |
| topic | teaching quality observation systems generalizability theory classroom observation validity |
| url | https://www.frontiersin.org/articles/10.3389/feduc.2025.1547204/full |
| work_keys_str_mv | AT arminjentsch usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores AT bassenden usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores AT naniteig usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores AT trudenilsen usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores AT johanneskonig usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores AT gabrielekaiser usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores AT gabrielekaiser usinggeneralizabilitytheorytoexaminetheeffectsofpresentationorderonclassroomobservationscores |