Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology
Abstract Background In Japan, reporting of medical device malfunctions and related health problems is mandatory, and efforts are being made to standardize terminology through the Adverse Event Terminology Collection of the Japan Federation of Medical Device Associations (JFMDA). Internationally, the...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2025-02-01
|
| Series: | BMC Medical Informatics and Decision Making |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12911-025-02912-0 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1823862000982163456 |
|---|---|
| author | Ayako Yagahara Masahito Uesugi Hideto Yokoi |
| author_facet | Ayako Yagahara Masahito Uesugi Hideto Yokoi |
| author_sort | Ayako Yagahara |
| collection | DOAJ |
| description | Abstract Background In Japan, reporting of medical device malfunctions and related health problems is mandatory, and efforts are being made to standardize terminology through the Adverse Event Terminology Collection of the Japan Federation of Medical Device Associations (JFMDA). Internationally, the Adverse Event Terminology of the International Medical Device Regulators Forum (IMDRF-AET) provides a standardized terminology collection in English. Mapping between the JFMDA terminology collection and the IMDRF-AET is critical to international harmonization. However, the process of translating the terminology collections from English to Japanese and reconciling them is done manually, resulting in high human workloads and potential inaccuracies. Objective The purpose of this study is to investigate the optimal machine translation model for the IMDRF-AET into Japanese for the part of a function for the automatic terminology mapping system. Methods English-Japanese parallel data for IMDRF-AET published by the Ministry of Health, Labor and Welfare in Japan was obtained from 50 sentences randomly extracted from the terms and their definitions. These English sentences were fed into the following machine translation models to produce Japanese translations: mBART50, m2m-100, Google Translation, Multilingual T5, GPT-3, ChatGPT, and GPT-4. The evaluations included the quantitative metrics of BiLingual Evaluation Understudy (BLEU), Character Error Rate (CER), Word Error Rate (WER), Metric for Evaluation of Translation with Explicit ORdering (METEOR), and Bidirectional Encoder Representations from Transformers (BERT) score, as well as qualitative evaluations by four experts. Results GPT-4 outperformed other models in both the quantitative and qualitative evaluations, with ChatGPT showing the same capability, but with lower quantitative scores, in the qualitative evaluation. Scores of other models, including mBART50 and m2m-100, lagged behind, particularly in the CER and BERT scores. Conclusion GPT-4’s superior performance in translating medical terminology, indicates its potential utility in improving the efficiency of the terminology mapping system. |
| format | Article |
| id | doaj-art-c8d30f53b9284524ac0ecb3daaf88e0d |
| institution | Kabale University |
| issn | 1472-6947 |
| language | English |
| publishDate | 2025-02-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Medical Informatics and Decision Making |
| spelling | doaj-art-c8d30f53b9284524ac0ecb3daaf88e0d2025-02-09T12:40:22ZengBMCBMC Medical Informatics and Decision Making1472-69472025-02-012511810.1186/s12911-025-02912-0Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminologyAyako Yagahara0Masahito Uesugi1Hideto Yokoi2Faculty of Health Sciences, Hokkaido University of ScienceFaculty of Medical Management and Informatics, Hokkaido Information UniversityDepartment of medical informatics, Kagawa University HospitalAbstract Background In Japan, reporting of medical device malfunctions and related health problems is mandatory, and efforts are being made to standardize terminology through the Adverse Event Terminology Collection of the Japan Federation of Medical Device Associations (JFMDA). Internationally, the Adverse Event Terminology of the International Medical Device Regulators Forum (IMDRF-AET) provides a standardized terminology collection in English. Mapping between the JFMDA terminology collection and the IMDRF-AET is critical to international harmonization. However, the process of translating the terminology collections from English to Japanese and reconciling them is done manually, resulting in high human workloads and potential inaccuracies. Objective The purpose of this study is to investigate the optimal machine translation model for the IMDRF-AET into Japanese for the part of a function for the automatic terminology mapping system. Methods English-Japanese parallel data for IMDRF-AET published by the Ministry of Health, Labor and Welfare in Japan was obtained from 50 sentences randomly extracted from the terms and their definitions. These English sentences were fed into the following machine translation models to produce Japanese translations: mBART50, m2m-100, Google Translation, Multilingual T5, GPT-3, ChatGPT, and GPT-4. The evaluations included the quantitative metrics of BiLingual Evaluation Understudy (BLEU), Character Error Rate (CER), Word Error Rate (WER), Metric for Evaluation of Translation with Explicit ORdering (METEOR), and Bidirectional Encoder Representations from Transformers (BERT) score, as well as qualitative evaluations by four experts. Results GPT-4 outperformed other models in both the quantitative and qualitative evaluations, with ChatGPT showing the same capability, but with lower quantitative scores, in the qualitative evaluation. Scores of other models, including mBART50 and m2m-100, lagged behind, particularly in the CER and BERT scores. Conclusion GPT-4’s superior performance in translating medical terminology, indicates its potential utility in improving the efficiency of the terminology mapping system.https://doi.org/10.1186/s12911-025-02912-0Machine translationDeep learningMedical device adverse event terminology |
| spellingShingle | Ayako Yagahara Masahito Uesugi Hideto Yokoi Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology BMC Medical Informatics and Decision Making Machine translation Deep learning Medical device adverse event terminology |
| title | Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology |
| title_full | Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology |
| title_fullStr | Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology |
| title_full_unstemmed | Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology |
| title_short | Exploration of the optimal deep learning model for english-Japanese machine translation of medical device adverse event terminology |
| title_sort | exploration of the optimal deep learning model for english japanese machine translation of medical device adverse event terminology |
| topic | Machine translation Deep learning Medical device adverse event terminology |
| url | https://doi.org/10.1186/s12911-025-02912-0 |
| work_keys_str_mv | AT ayakoyagahara explorationoftheoptimaldeeplearningmodelforenglishjapanesemachinetranslationofmedicaldeviceadverseeventterminology AT masahitouesugi explorationoftheoptimaldeeplearningmodelforenglishjapanesemachinetranslationofmedicaldeviceadverseeventterminology AT hidetoyokoi explorationoftheoptimaldeeplearningmodelforenglishjapanesemachinetranslationofmedicaldeviceadverseeventterminology |