Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories
We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover cla...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
American Physical Society
2025-07-01
|
| Series: | Physical Review Physics Education Research |
| Online Access: | http://doi.org/10.1103/98hg-rkrf |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850100641565769728 |
|---|---|
| author | Gerd Kortemeyer Marina Babayeva Giulia Polverini Ralf Widenhorn Bor Gregorcic |
| author_facet | Gerd Kortemeyer Marina Babayeva Giulia Polverini Ralf Widenhorn Bor Gregorcic |
| author_sort | Gerd Kortemeyer |
| collection | DOAJ |
| description | We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics, electromagnetism, optics, and thermodynamics, as well as relativity, quantum mechanics, astronomy, mathematics, and laboratory skills. Unlike previous text-only studies, we uploaded the inventories as images to reflect what a student would see on paper, thereby assessing the system’s multimodal functionality. Our results indicate variation in performance across subjects, with laboratory skills standing out as the weakest. We also observe differences across languages, with English and European languages showing the strongest performance. Notably, the relative difficulty of an inventory item is largely independent of the language of the test. When comparing AI results to existing literature on student performance, we find that the AI system outperforms average postinstruction undergraduate students in all subject categories except laboratory skills. Furthermore, the AI performs worse on items requiring visual interpretation of images than on those that are purely text-based. While our exploratory findings show GPT-4o’s potential usefulness in physics education, they highlight the critical need for instructors to foster students’ ability to critically evaluate AI outputs, adapt curricula thoughtfully in response to AI advancements, and address equity concerns associated with AI integration. |
| format | Article |
| id | doaj-art-ec2e2ade42574beeb9717c66027c2992 |
| institution | DOAJ |
| issn | 2469-9896 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | American Physical Society |
| record_format | Article |
| series | Physical Review Physics Education Research |
| spelling | doaj-art-ec2e2ade42574beeb9717c66027c29922025-08-20T02:40:14ZengAmerican Physical SocietyPhysical Review Physics Education Research2469-98962025-07-0121202010110.1103/98hg-rkrfMultilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventoriesGerd KortemeyerMarina BabayevaGiulia PolveriniRalf WidenhornBor GregorcicWe investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics, electromagnetism, optics, and thermodynamics, as well as relativity, quantum mechanics, astronomy, mathematics, and laboratory skills. Unlike previous text-only studies, we uploaded the inventories as images to reflect what a student would see on paper, thereby assessing the system’s multimodal functionality. Our results indicate variation in performance across subjects, with laboratory skills standing out as the weakest. We also observe differences across languages, with English and European languages showing the strongest performance. Notably, the relative difficulty of an inventory item is largely independent of the language of the test. When comparing AI results to existing literature on student performance, we find that the AI system outperforms average postinstruction undergraduate students in all subject categories except laboratory skills. Furthermore, the AI performs worse on items requiring visual interpretation of images than on those that are purely text-based. While our exploratory findings show GPT-4o’s potential usefulness in physics education, they highlight the critical need for instructors to foster students’ ability to critically evaluate AI outputs, adapt curricula thoughtfully in response to AI advancements, and address equity concerns associated with AI integration.http://doi.org/10.1103/98hg-rkrf |
| spellingShingle | Gerd Kortemeyer Marina Babayeva Giulia Polverini Ralf Widenhorn Bor Gregorcic Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories Physical Review Physics Education Research |
| title | Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories |
| title_full | Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories |
| title_fullStr | Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories |
| title_full_unstemmed | Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories |
| title_short | Multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories |
| title_sort | multilingual performance of a multimodal artificial intelligence system on multisubject physics concept inventories |
| url | http://doi.org/10.1103/98hg-rkrf |
| work_keys_str_mv | AT gerdkortemeyer multilingualperformanceofamultimodalartificialintelligencesystemonmultisubjectphysicsconceptinventories AT marinababayeva multilingualperformanceofamultimodalartificialintelligencesystemonmultisubjectphysicsconceptinventories AT giuliapolverini multilingualperformanceofamultimodalartificialintelligencesystemonmultisubjectphysicsconceptinventories AT ralfwidenhorn multilingualperformanceofamultimodalartificialintelligencesystemonmultisubjectphysicsconceptinventories AT borgregorcic multilingualperformanceofamultimodalartificialintelligencesystemonmultisubjectphysicsconceptinventories |