Developing and validating measures for AI literacy tests: From self-reported to objective measures
The majority of AI literacy studies have designed and developed self-reported questionnaires to assess AI learning and understanding. These studies assessed students' perceived AI capability rather than AI literacy because self-perceptions are seldom an accurate account of true measures. Intern...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Elsevier
2024-12-01
|
| Series: | Computers and Education: Artificial Intelligence |
| Subjects: | |
| Online Access: | http://www.sciencedirect.com/science/article/pii/S2666920X24000857 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850053648544956416 |
|---|---|
| author | Thomas K.F. Chiu Yifan Chen King Woon Yau Ching-sing Chai Helen Meng Irwin King Savio Wong Yeung Yam |
| author_facet | Thomas K.F. Chiu Yifan Chen King Woon Yau Ching-sing Chai Helen Meng Irwin King Savio Wong Yeung Yam |
| author_sort | Thomas K.F. Chiu |
| collection | DOAJ |
| description | The majority of AI literacy studies have designed and developed self-reported questionnaires to assess AI learning and understanding. These studies assessed students' perceived AI capability rather than AI literacy because self-perceptions are seldom an accurate account of true measures. International assessment programs that use objective measures to assess science, mathematical, digital, and computational literacy back up this argument. Furthermore, because AI education research is still in its infancy, the current definition of AI literacy in the literature may not meet the needs of young students. Therefore, this study aims to develop and validate an AI literacy test for school students within the interdisciplinary project known as AI4future. Engineering and education researchers created and selected 25 multiple-choice questions to accomplish this goal, and school teachers validated them while developing an AI curriculum for middle schools. 2390 students in grades 7 to 9 took the test. We used a Rasch model to investigate the discrimination, reliability, and validity of the items. The results showed that the model met the unidimensionality assumption and demonstrated a set of reliable and valid items. They indicate the quality of the test items. The test enables AI education researchers and practitioners to appropriately evaluate their AI-related education interventions. |
| format | Article |
| id | doaj-art-17f01103fd554fb4ad45da29958eb812 |
| institution | DOAJ |
| issn | 2666-920X |
| language | English |
| publishDate | 2024-12-01 |
| publisher | Elsevier |
| record_format | Article |
| series | Computers and Education: Artificial Intelligence |
| spelling | doaj-art-17f01103fd554fb4ad45da29958eb8122025-08-20T02:52:28ZengElsevierComputers and Education: Artificial Intelligence2666-920X2024-12-01710028210.1016/j.caeai.2024.100282Developing and validating measures for AI literacy tests: From self-reported to objective measuresThomas K.F. Chiu0Yifan Chen1King Woon Yau2Ching-sing Chai3Helen Meng4Irwin King5Savio Wong6Yeung Yam7Department of Curriculum and Instruction Faculty of Education and Centre for Learning Sciences and Technologies And, Centre for University and School Partnership the Chinese University of Hong Kong, Shatin, NT, SAR, Hong Kong; Corresponding author.Faculty of Engineering, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongFaculty of Engineering, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongDepartment of Curriculum and Instruction, Associate Dean (Postgraduate Studies), Faculty of Education, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongDepartment of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongDepartment of Computer Science & Engineering, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongDepartment of Educational Psychology, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongDepartment of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, NT, SAR, Hong KongThe majority of AI literacy studies have designed and developed self-reported questionnaires to assess AI learning and understanding. These studies assessed students' perceived AI capability rather than AI literacy because self-perceptions are seldom an accurate account of true measures. International assessment programs that use objective measures to assess science, mathematical, digital, and computational literacy back up this argument. Furthermore, because AI education research is still in its infancy, the current definition of AI literacy in the literature may not meet the needs of young students. Therefore, this study aims to develop and validate an AI literacy test for school students within the interdisciplinary project known as AI4future. Engineering and education researchers created and selected 25 multiple-choice questions to accomplish this goal, and school teachers validated them while developing an AI curriculum for middle schools. 2390 students in grades 7 to 9 took the test. We used a Rasch model to investigate the discrimination, reliability, and validity of the items. The results showed that the model met the unidimensionality assumption and demonstrated a set of reliable and valid items. They indicate the quality of the test items. The test enables AI education researchers and practitioners to appropriately evaluate their AI-related education interventions.http://www.sciencedirect.com/science/article/pii/S2666920X24000857AI literacyInstrumentK-12 educationAI educationCo-design processMeasures |
| spellingShingle | Thomas K.F. Chiu Yifan Chen King Woon Yau Ching-sing Chai Helen Meng Irwin King Savio Wong Yeung Yam Developing and validating measures for AI literacy tests: From self-reported to objective measures Computers and Education: Artificial Intelligence AI literacy Instrument K-12 education AI education Co-design process Measures |
| title | Developing and validating measures for AI literacy tests: From self-reported to objective measures |
| title_full | Developing and validating measures for AI literacy tests: From self-reported to objective measures |
| title_fullStr | Developing and validating measures for AI literacy tests: From self-reported to objective measures |
| title_full_unstemmed | Developing and validating measures for AI literacy tests: From self-reported to objective measures |
| title_short | Developing and validating measures for AI literacy tests: From self-reported to objective measures |
| title_sort | developing and validating measures for ai literacy tests from self reported to objective measures |
| topic | AI literacy Instrument K-12 education AI education Co-design process Measures |
| url | http://www.sciencedirect.com/science/article/pii/S2666920X24000857 |
| work_keys_str_mv | AT thomaskfchiu developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT yifanchen developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT kingwoonyau developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT chingsingchai developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT helenmeng developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT irwinking developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT saviowong developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures AT yeungyam developingandvalidatingmeasuresforailiteracytestsfromselfreportedtoobjectivemeasures |