A Model for Automated Business Writing Assessment
This study is aimed at building an automated model for business writing assessment, based on 14 rubrics that integrate EFL teacher assessment frameworks and identify expected performance against various criteria (including language, task fulfillment, content knowledge, register, format, and cohesion...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Yaroslavl State University
2022-12-01
|
| Series: | Моделирование и анализ информационных систем |
| Subjects: | |
| Online Access: | https://www.mais-journal.ru/jour/article/view/1751 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849688296053014528 |
|---|---|
| author | Daniil Dmitrievich Zafievsky Nadezhda Stanislavona Lagutina Oksana Andreyevna Melnikova Anatoliy Yurievich Poletaev |
| author_facet | Daniil Dmitrievich Zafievsky Nadezhda Stanislavona Lagutina Oksana Andreyevna Melnikova Anatoliy Yurievich Poletaev |
| author_sort | Daniil Dmitrievich Zafievsky |
| collection | DOAJ |
| description | This study is aimed at building an automated model for business writing assessment, based on 14 rubrics that integrate EFL teacher assessment frameworks and identify expected performance against various criteria (including language, task fulfillment, content knowledge, register, format, and cohesion). We developed algorithms for determining the corresponding numerical features using methods and tools for automatic text analysis. The algorithms are based on a syntactic analysis with the use of dictionaries. The model performance was subsequently evaluated on a corpus of 20 teacher-assessed business letters. Heat maps and UMAP results represent comparison between teachers’ and automated score reports. Results showed no significant discrepancies between teachers’ and automated score reports, yet detected bias in teachers’ reports. Findings suggest that the developed model has proved to be an efficient tool for natural language processing with high interpretability of the results, the roadmap for further improvement and a valid and unbiased alternative to teachers’ assessment. The results may lay the groundwork for developing an automatic students’ language profile. Although the model was specifically designed for business letter assessment, it can be easily adapted for assessing other writing tasks, e.g. by replacing dictionaries. |
| format | Article |
| id | doaj-art-4db076bc41ea42dfb1a7b2ad3b17de10 |
| institution | DOAJ |
| issn | 1818-1015 2313-5417 |
| language | English |
| publishDate | 2022-12-01 |
| publisher | Yaroslavl State University |
| record_format | Article |
| series | Моделирование и анализ информационных систем |
| spelling | doaj-art-4db076bc41ea42dfb1a7b2ad3b17de102025-08-20T03:22:03ZengYaroslavl State UniversityМоделирование и анализ информационных систем1818-10152313-54172022-12-0129434836510.18255/1818-1015-2022-4-348-3651356A Model for Automated Business Writing AssessmentDaniil Dmitrievich Zafievsky0Nadezhda Stanislavona Lagutina1Oksana Andreyevna Melnikova2Anatoliy Yurievich Poletaev3P. G. Demidov Yaroslavl State UniversityP. G. Demidov Yaroslavl State UniversityP. G. Demidov Yaroslavl State UniversityP. G. Demidov Yaroslavl State UniversityThis study is aimed at building an automated model for business writing assessment, based on 14 rubrics that integrate EFL teacher assessment frameworks and identify expected performance against various criteria (including language, task fulfillment, content knowledge, register, format, and cohesion). We developed algorithms for determining the corresponding numerical features using methods and tools for automatic text analysis. The algorithms are based on a syntactic analysis with the use of dictionaries. The model performance was subsequently evaluated on a corpus of 20 teacher-assessed business letters. Heat maps and UMAP results represent comparison between teachers’ and automated score reports. Results showed no significant discrepancies between teachers’ and automated score reports, yet detected bias in teachers’ reports. Findings suggest that the developed model has proved to be an efficient tool for natural language processing with high interpretability of the results, the roadmap for further improvement and a valid and unbiased alternative to teachers’ assessment. The results may lay the groundwork for developing an automatic students’ language profile. Although the model was specifically designed for business letter assessment, it can be easily adapted for assessing other writing tasks, e.g. by replacing dictionaries.https://www.mais-journal.ru/jour/article/view/1751natural language processingtext featuresautomated essay scoringbusiness letter |
| spellingShingle | Daniil Dmitrievich Zafievsky Nadezhda Stanislavona Lagutina Oksana Andreyevna Melnikova Anatoliy Yurievich Poletaev A Model for Automated Business Writing Assessment Моделирование и анализ информационных систем natural language processing text features automated essay scoring business letter |
| title | A Model for Automated Business Writing Assessment |
| title_full | A Model for Automated Business Writing Assessment |
| title_fullStr | A Model for Automated Business Writing Assessment |
| title_full_unstemmed | A Model for Automated Business Writing Assessment |
| title_short | A Model for Automated Business Writing Assessment |
| title_sort | model for automated business writing assessment |
| topic | natural language processing text features automated essay scoring business letter |
| url | https://www.mais-journal.ru/jour/article/view/1751 |
| work_keys_str_mv | AT daniildmitrievichzafievsky amodelforautomatedbusinesswritingassessment AT nadezhdastanislavonalagutina amodelforautomatedbusinesswritingassessment AT oksanaandreyevnamelnikova amodelforautomatedbusinesswritingassessment AT anatoliyyurievichpoletaev amodelforautomatedbusinesswritingassessment AT daniildmitrievichzafievsky modelforautomatedbusinesswritingassessment AT nadezhdastanislavonalagutina modelforautomatedbusinesswritingassessment AT oksanaandreyevnamelnikova modelforautomatedbusinesswritingassessment AT anatoliyyurievichpoletaev modelforautomatedbusinesswritingassessment |