Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention
Artificial Intelligence (AI)-based systems have been proposed to aid Mental Health Professionals (MHPs) in various tasks, including the prevention of suicide by identifying Suicidal Ideation (SI). However, these systems may lack transparency and thereby create mistrust among MHPs. Explainable Artifi...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10945851/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850185216110362624 |
|---|---|
| author | Adonias Caetano de Oliveira Joao Pedro Cavalcanti Azevedo Livia Ruback Rayele Moreira Silmar Silva Teixeira Ariel Soares Teles |
| author_facet | Adonias Caetano de Oliveira Joao Pedro Cavalcanti Azevedo Livia Ruback Rayele Moreira Silmar Silva Teixeira Ariel Soares Teles |
| author_sort | Adonias Caetano de Oliveira |
| collection | DOAJ |
| description | Artificial Intelligence (AI)-based systems have been proposed to aid Mental Health Professionals (MHPs) in various tasks, including the prevention of suicide by identifying Suicidal Ideation (SI). However, these systems may lack transparency and thereby create mistrust among MHPs. Explainable Artificial Intelligence (XAI) methods can elucidate how features influence system predictions, aiding MHPs in understanding them. This exploratory study aims to investigate how MHPs’ trust is influenced by AI explanations (educational intervention and XAI methods) and other factors (professional background, knowledge of AI and computing, and reported system misclassification). We conducted an experiment using Boamente, an AI-powered clinical decision support system designed to assist MHPs in suicide prevention. Boamente identifies SI in Brazilian Portuguese texts typed on smartphones by leveraging a Large Language Model (LLM) for analysis. The results demonstrate that professional background, knowledge of AI and computing, and educational intervention had no statistically significant effect on trust. In contrast, trust was affected by factors such as LLM prediction explanations, the quality of explanations, and reported misclassification. Therefore, providing prediction explanations to understand the inner workings of AI models led MHPs to be more critical in relation to predictions, while there was an overtrust on MHPs when no explanations were provided. Furthermore, disagreement with LLM classifications and perceptions of system vulnerabilities also affected trust. |
| format | Article |
| id | doaj-art-a77aae1dd810498b89ea935bb65752b8 |
| institution | OA Journals |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-a77aae1dd810498b89ea935bb65752b82025-08-20T02:16:49ZengIEEEIEEE Access2169-35362025-01-0113609876100510.1109/ACCESS.2025.355624510945851Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide PreventionAdonias Caetano de Oliveira0https://orcid.org/0000-0002-5643-2916Joao Pedro Cavalcanti Azevedo1https://orcid.org/0009-0003-2035-6872Livia Ruback2https://orcid.org/0000-0001-5000-2280Rayele Moreira3https://orcid.org/0000-0002-4480-732XSilmar Silva Teixeira4https://orcid.org/0000-0002-9240-1228Ariel Soares Teles5https://orcid.org/0000-0002-0840-3870Programa de Pós-Graduação em Biotecnologia (PPGBiotec), Universidade Federal do Delta do Parnaíba, Parnaíba, BrazilPrograma de Pós-graduação em Ciência da Computação (PPGCC), Universidade Federal do Maranhão, São Luís, BrazilFaculdade de Tecnologia (FT), Universidade Estadual de Campinas, Campinas, BrazilPrograma de Pós-Graduação em Biotecnologia (PPGBiotec), Universidade Federal do Delta do Parnaíba, Parnaíba, BrazilPrograma de Pós-Graduação em Biotecnologia (PPGBiotec), Universidade Federal do Delta do Parnaíba, Parnaíba, BrazilPrograma de Pós-Graduação em Biotecnologia (PPGBiotec), Universidade Federal do Delta do Parnaíba, Parnaíba, BrazilArtificial Intelligence (AI)-based systems have been proposed to aid Mental Health Professionals (MHPs) in various tasks, including the prevention of suicide by identifying Suicidal Ideation (SI). However, these systems may lack transparency and thereby create mistrust among MHPs. Explainable Artificial Intelligence (XAI) methods can elucidate how features influence system predictions, aiding MHPs in understanding them. This exploratory study aims to investigate how MHPs’ trust is influenced by AI explanations (educational intervention and XAI methods) and other factors (professional background, knowledge of AI and computing, and reported system misclassification). We conducted an experiment using Boamente, an AI-powered clinical decision support system designed to assist MHPs in suicide prevention. Boamente identifies SI in Brazilian Portuguese texts typed on smartphones by leveraging a Large Language Model (LLM) for analysis. The results demonstrate that professional background, knowledge of AI and computing, and educational intervention had no statistically significant effect on trust. In contrast, trust was affected by factors such as LLM prediction explanations, the quality of explanations, and reported misclassification. Therefore, providing prediction explanations to understand the inner workings of AI models led MHPs to be more critical in relation to predictions, while there was an overtrust on MHPs when no explanations were provided. Furthermore, disagreement with LLM classifications and perceptions of system vulnerabilities also affected trust.https://ieeexplore.ieee.org/document/10945851/Explainable artificial intelligencetrustmedical AImental healthsuicidedigital phenotyping |
| spellingShingle | Adonias Caetano de Oliveira Joao Pedro Cavalcanti Azevedo Livia Ruback Rayele Moreira Silmar Silva Teixeira Ariel Soares Teles Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention IEEE Access Explainable artificial intelligence trust medical AI mental health suicide digital phenotyping |
| title | Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention |
| title_full | Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention |
| title_fullStr | Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention |
| title_full_unstemmed | Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention |
| title_short | Effect of Explainable Artificial Intelligence on Trust of Mental Health Professionals in an AI-Based System for Suicide Prevention |
| title_sort | effect of explainable artificial intelligence on trust of mental health professionals in an ai based system for suicide prevention |
| topic | Explainable artificial intelligence trust medical AI mental health suicide digital phenotyping |
| url | https://ieeexplore.ieee.org/document/10945851/ |
| work_keys_str_mv | AT adoniascaetanodeoliveira effectofexplainableartificialintelligenceontrustofmentalhealthprofessionalsinanaibasedsystemforsuicideprevention AT joaopedrocavalcantiazevedo effectofexplainableartificialintelligenceontrustofmentalhealthprofessionalsinanaibasedsystemforsuicideprevention AT liviaruback effectofexplainableartificialintelligenceontrustofmentalhealthprofessionalsinanaibasedsystemforsuicideprevention AT rayelemoreira effectofexplainableartificialintelligenceontrustofmentalhealthprofessionalsinanaibasedsystemforsuicideprevention AT silmarsilvateixeira effectofexplainableartificialintelligenceontrustofmentalhealthprofessionalsinanaibasedsystemforsuicideprevention AT arielsoaresteles effectofexplainableartificialintelligenceontrustofmentalhealthprofessionalsinanaibasedsystemforsuicideprevention |