Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS?
Introduction: With the growing development and adoption of artificial intelligence in healthcare and across other sectors of society, various user-friendly and engaging tools to support research have emerged, such as chatbots, notably ChatGPT. Objective: To investigate the performance of ChatGPT as...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2024-11-01
|
| Series: | Informatics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-9709/11/4/84 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850041546895785984 |
|---|---|
| author | Bernardo Nascimento Teixeira Ana Leitão Generosa Nascimento Adalberto Campos-Fernandes Francisco Cercas |
| author_facet | Bernardo Nascimento Teixeira Ana Leitão Generosa Nascimento Adalberto Campos-Fernandes Francisco Cercas |
| author_sort | Bernardo Nascimento Teixeira |
| collection | DOAJ |
| description | Introduction: With the growing development and adoption of artificial intelligence in healthcare and across other sectors of society, various user-friendly and engaging tools to support research have emerged, such as chatbots, notably ChatGPT. Objective: To investigate the performance of ChatGPT as an assistant to medical coders using the ICD-10-CM/PCS. Methodology: We conducted a prospective exploratory study between 2023 and 2024 over 6 months. A total of 150 clinical cases coded using the ICD-10-CM/PCS, extracted from technical coding books, were systematically randomized. All cases were translated into Portuguese (the native language of the authors) and English (the native language of the ICD-10-CM/PCS). These clinical cases varied in complexity levels regarding the quantity of diagnoses and procedures, as well as the nature of the clinical information. Each case was input into the 2023 ChatGPT free version. The coding obtained from ChatGPT was analyzed by a senior medical auditor/coder and compared with the expected results. Results: Regarding the correct codes, ChatGPT’s performance was higher by approximately 29 percentage points between diagnoses and procedures, with greater proficiency in diagnostic codes. The accuracy rate for codes was similar across languages, with rates of 31.0% and 31.9%. The error rate in procedure codes was substantially higher than that in diagnostic codes by almost four times. For missing information, a higher incidence was observed in diagnoses compared to procedures of slightly more than double the comparative rates. Additionally, there was a statistically significant excess of codes not related to clinical information, which was higher in procedures and nearly the same value in both languages under study. Conclusion: Given the ease of access to these tools, this investigation serves as an awareness factor, demonstrating that ChatGPT can assist the medical coder in directed research. However, it does not replace their technical validation in this process. Therefore, further developments of this tool are necessary to increase the quality and reliability of the results. |
| format | Article |
| id | doaj-art-e35ccde3581e49ba9c3acd61222c584d |
| institution | DOAJ |
| issn | 2227-9709 |
| language | English |
| publishDate | 2024-11-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Informatics |
| spelling | doaj-art-e35ccde3581e49ba9c3acd61222c584d2025-08-20T02:55:45ZengMDPI AGInformatics2227-97092024-11-011148410.3390/informatics11040084Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS?Bernardo Nascimento Teixeira0Ana Leitão1Generosa Nascimento2Adalberto Campos-Fernandes3Francisco Cercas4Iscte-Instituto Universitário de Lisboa (ISCTE-IUL), Av. das Forças Armadas, 1649-026 Lisboa, PortugalUnidade Local de Saúde de Lisboa Ocidental, E.P.E (ULSLO), Estrada do Forte do Alto do Duque, 1449-005 Lisboa, PortugalIscte-Instituto Universitário de Lisboa (ISCTE-IUL), Av. das Forças Armadas, 1649-026 Lisboa, PortugalNOVA National School of Public Health, Universidade Nova de Lisboa (ENSP-UNL), Av. Padre Cruz, 1600-407 Lisboa, PortugalIscte-Instituto Universitário de Lisboa (ISCTE-IUL), Av. das Forças Armadas, 1649-026 Lisboa, PortugalIntroduction: With the growing development and adoption of artificial intelligence in healthcare and across other sectors of society, various user-friendly and engaging tools to support research have emerged, such as chatbots, notably ChatGPT. Objective: To investigate the performance of ChatGPT as an assistant to medical coders using the ICD-10-CM/PCS. Methodology: We conducted a prospective exploratory study between 2023 and 2024 over 6 months. A total of 150 clinical cases coded using the ICD-10-CM/PCS, extracted from technical coding books, were systematically randomized. All cases were translated into Portuguese (the native language of the authors) and English (the native language of the ICD-10-CM/PCS). These clinical cases varied in complexity levels regarding the quantity of diagnoses and procedures, as well as the nature of the clinical information. Each case was input into the 2023 ChatGPT free version. The coding obtained from ChatGPT was analyzed by a senior medical auditor/coder and compared with the expected results. Results: Regarding the correct codes, ChatGPT’s performance was higher by approximately 29 percentage points between diagnoses and procedures, with greater proficiency in diagnostic codes. The accuracy rate for codes was similar across languages, with rates of 31.0% and 31.9%. The error rate in procedure codes was substantially higher than that in diagnostic codes by almost four times. For missing information, a higher incidence was observed in diagnoses compared to procedures of slightly more than double the comparative rates. Additionally, there was a statistically significant excess of codes not related to clinical information, which was higher in procedures and nearly the same value in both languages under study. Conclusion: Given the ease of access to these tools, this investigation serves as an awareness factor, demonstrating that ChatGPT can assist the medical coder in directed research. However, it does not replace their technical validation in this process. Therefore, further developments of this tool are necessary to increase the quality and reliability of the results.https://www.mdpi.com/2227-9709/11/4/84ChatGPTartificial intelligenceICD-10-CM/PCSclinical coding |
| spellingShingle | Bernardo Nascimento Teixeira Ana Leitão Generosa Nascimento Adalberto Campos-Fernandes Francisco Cercas Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS? Informatics ChatGPT artificial intelligence ICD-10-CM/PCS clinical coding |
| title | Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS? |
| title_full | Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS? |
| title_fullStr | Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS? |
| title_full_unstemmed | Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS? |
| title_short | Can ChatGPT Support Clinical Coding Using the ICD-10-CM/PCS? |
| title_sort | can chatgpt support clinical coding using the icd 10 cm pcs |
| topic | ChatGPT artificial intelligence ICD-10-CM/PCS clinical coding |
| url | https://www.mdpi.com/2227-9709/11/4/84 |
| work_keys_str_mv | AT bernardonascimentoteixeira canchatgptsupportclinicalcodingusingtheicd10cmpcs AT analeitao canchatgptsupportclinicalcodingusingtheicd10cmpcs AT generosanascimento canchatgptsupportclinicalcodingusingtheicd10cmpcs AT adalbertocamposfernandes canchatgptsupportclinicalcodingusingtheicd10cmpcs AT franciscocercas canchatgptsupportclinicalcodingusingtheicd10cmpcs |