Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill
In the process of measuring and assessing high-level cognitive skills, interference of rater errors in measurements brings about a constant concern and low objectivity. The main purpose of this study was to investigate the impact of rater training on rater errors in the process of assessing individu...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
izzet kara
2022-06-01
|
| Series: | International Journal of Assessment Tools in Education |
| Subjects: | |
| Online Access: | https://dergipark.org.tr/tr/download/article-file/1565419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850067884481445888 |
|---|---|
| author | İsmail Karakaya Mehmet Şata |
| author_facet | İsmail Karakaya Mehmet Şata |
| author_sort | İsmail Karakaya |
| collection | DOAJ |
| description | In the process of measuring and assessing high-level cognitive skills, interference of rater errors in measurements brings about a constant concern and low objectivity. The main purpose of this study was to investigate the impact of rater training on rater errors in the process of assessing individual performance. The study was conducted with a pretest-posttest control group quasi-experimental design. In this research, 45 raters were employed, 23 from the control group and 22 from the experimental group. As data collection tools, a writing task that was developed by IELTS and an analytical rubric that was developed to assess academic writing skills were used. As part of the experimental procedure, rater training was provided and this training was implemented by combining rater error training and frame of reference training. When the findings of the study were examined, it was found that the control and experimental groups were similar to each other before the experiment, however, after the experimental process, the study group made more valid and reliable measurements. As a result, it was investigated that the rater training given had an impact on rater errors such as rater severity, rater leniency, central tendency, and Halo effect. Based on the obtained findings, some suggestions were offered for researchers and future studies. |
| format | Article |
| id | doaj-art-7aeaa43f78604607a8eb0de684d02abc |
| institution | DOAJ |
| issn | 2148-7456 |
| language | English |
| publishDate | 2022-06-01 |
| publisher | izzet kara |
| record_format | Article |
| series | International Journal of Assessment Tools in Education |
| spelling | doaj-art-7aeaa43f78604607a8eb0de684d02abc2025-08-20T02:48:12Zengizzet karaInternational Journal of Assessment Tools in Education2148-74562022-06-019249251410.21449/ijate.877035618Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skillİsmail Karakaya0https://orcid.org/0000-0003-4308-6919Mehmet Şata1https://orcid.org/0000-0003-2683-4997Gazı UniversityAgri Ibrahim Cecen UniversityIn the process of measuring and assessing high-level cognitive skills, interference of rater errors in measurements brings about a constant concern and low objectivity. The main purpose of this study was to investigate the impact of rater training on rater errors in the process of assessing individual performance. The study was conducted with a pretest-posttest control group quasi-experimental design. In this research, 45 raters were employed, 23 from the control group and 22 from the experimental group. As data collection tools, a writing task that was developed by IELTS and an analytical rubric that was developed to assess academic writing skills were used. As part of the experimental procedure, rater training was provided and this training was implemented by combining rater error training and frame of reference training. When the findings of the study were examined, it was found that the control and experimental groups were similar to each other before the experiment, however, after the experimental process, the study group made more valid and reliable measurements. As a result, it was investigated that the rater training given had an impact on rater errors such as rater severity, rater leniency, central tendency, and Halo effect. Based on the obtained findings, some suggestions were offered for researchers and future studies.https://dergipark.org.tr/tr/download/article-file/1565419rater trainingrater errorsmany facet rasch modelvalidityreliabilityrater trainingrater errorsmany facet rasch modelvalidityreliability |
| spellingShingle | İsmail Karakaya Mehmet Şata Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill International Journal of Assessment Tools in Education rater training rater errors many facet rasch model validity reliability rater training rater errors many facet rasch model validity reliability |
| title | Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill |
| title_full | Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill |
| title_fullStr | Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill |
| title_full_unstemmed | Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill |
| title_short | Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill |
| title_sort | investigating the impact of rater training on rater errors in the process of assessing writing skill |
| topic | rater training rater errors many facet rasch model validity reliability rater training rater errors many facet rasch model validity reliability |
| url | https://dergipark.org.tr/tr/download/article-file/1565419 |
| work_keys_str_mv | AT ismailkarakaya investigatingtheimpactofratertrainingonratererrorsintheprocessofassessingwritingskill AT mehmetsata investigatingtheimpactofratertrainingonratererrorsintheprocessofassessingwritingskill |