Models for analyzing the complexity of English words in the text on the scale from A1 to C2 - М. Бєліков; Т. Ліхоузова; Ю. Олійник (Adaptivni Sistemi Avtomatičnogo Upravlinnâ)
At the current stage of globalization, English plays a key role as the language of international communication. This leads to the fact that more and more people become its carriers at various levels. The work is devoted to the analysis of English words on the scale from A1 to C2, which corresponds to the lowest and highest levels of proficiency according to the CEFR standards. A model that predicts the difficulty of words in a text can be used to improve the educational process. For example, it is possible to find a list of likely unknown and difficult words for the end user in any text depending on his level of English language proficiency. This approach will facilitate the language learning process by providing a personalized list of words to focus on. Also, the model can be useful for analyzing the complexity of texts depending on the number of words of each level of complexity in them. This can help teachers prepare materials that match their students' level of knowledge and identify words that may be difficult for them to understand. SQLite database of English words and the number of their occurrences in English books from 1900 to 2019 has been developed. The functionality is implemented using SQL scripts. Python programming language was used to write ETL processes, data analysis, creation, training and comparison of word complexity prediction models, Sqlite3, Lemminflect, NumPy, Seaborn, Matplotlib, SciPy, Sklearn, SpaCy and XGBoost libraries were used. An application in the Python programming language is proposed, which receives a sample of data from the created storage, displays them graphically, performs intellectual analysis, trains and compares models according to accuracy, precision, recall and f1-score metrics. To analyze the data and predict the difficulty level of English words according to the CEFR scale from A1 to C2 based on their frequency of occurrence in the English language, the following models were used: PchipInterpolator, logarithmic model, Gradient Boosting Regressor, Random Forest Regressor and XGB regressor. The results of each model were evaluated on the test sample, and the best model was chosen to further predict the difficulty level of all other English words.