How accurately can supervised machine learning model predict a targeted psychiatric disorder?
Abstract Background Hoarding disorder (HD) is characterized by a compulsion to collect belongings, and to experience significant distress when parting from them. HD is often misdiagnosed for several reasons. These include patient and family lack of recognition that it is a psychiatric disorder and p...
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
BMC
2024-10-01
|
| Series: | BMC Psychiatry |
| Subjects: | |
| Online Access: | https://doi.org/10.1186/s12888-024-06152-w |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850181656691867648 |
|---|---|
| author | Haitham Jahrami Amir H. Pakpour Waqar Husain Achraf Ammar Zahra Saif Ali Husain Alsalman Adel Aloffi Khaled Trabelsi Seithikurippu R. Pandi-Perumal Michael V. Vitiello |
| author_facet | Haitham Jahrami Amir H. Pakpour Waqar Husain Achraf Ammar Zahra Saif Ali Husain Alsalman Adel Aloffi Khaled Trabelsi Seithikurippu R. Pandi-Perumal Michael V. Vitiello |
| author_sort | Haitham Jahrami |
| collection | DOAJ |
| description | Abstract Background Hoarding disorder (HD) is characterized by a compulsion to collect belongings, and to experience significant distress when parting from them. HD is often misdiagnosed for several reasons. These include patient and family lack of recognition that it is a psychiatric disorder and professionals’ lack of relevant expertise with it. This study evaluates the ability of a supervised machine learning (ML) model to match the diagnostic skills of psychiatrists when presented with equivalent information pertinent to symptoms of HD. Methods Five hundred online participants were randomly recruited and completed the Hoarding Rating Scale-Self Report (HRS-SR) and the Generalized Anxiety Disorder 7-item (GAD-7) scale. Responses to the questionnaires were read by an ML model. Responses to the HRS-SR were then converted into anonymized, random-equivalent texts. Each of these individual texts was presented in random order to two experienced psychiatrists who were independently asked for a provisional diagnosis - e.g.; the presence or absence of HD. In case of disagreement between the two assessors, a third psychiatrist broke the tie. A decision tree classification model was employed to predict clinical HD using self-report data from two psychological tests, the HRS-SR and GAD-7. The target variable was whether a participant had clinical HD, while the predictive variables were the continuous scores from the HRS-SR and GAD-7 tests. The model’s performance was evaluated using a confusion matrix, which compared the observed diagnoses with the predicted diagnoses to assess accuracy. Results According to the psychiatrists, approximately 10% of the participants fulfilled DSM-5 diagnostic criteria for HD. 93% of the clinician-identified cases were identified by the ML model based on HRS-SR and GAD-7 scores. A decision tree plot model demonstrated that about 60% of the cases could be detected by the HRS-SR alone while the rest required a combination of HRS-SR and GAD-7 scores. ML evaluation metrics showed satisfactory performance, with a Matthews Correlation Coefficient of 55%; Area Under Curve (AUC), 79%; a Negative Predictive Value of 76%; and a False Negative Rate of 24%. Conclusions Study findings strongly suggest that ML can, in the future, play a significant role in the risk assessment of psychiatric disorders prior to face-to-face consultation. By using AI to scan big data questionnaire responses, wait time for seriously ill patients can be substantially cut, and prognoses substantially improved. |
| format | Article |
| id | doaj-art-7add043437194b0c8cd83a6ed6255ca2 |
| institution | OA Journals |
| issn | 1471-244X |
| language | English |
| publishDate | 2024-10-01 |
| publisher | BMC |
| record_format | Article |
| series | BMC Psychiatry |
| spelling | doaj-art-7add043437194b0c8cd83a6ed6255ca22025-08-20T02:17:50ZengBMCBMC Psychiatry1471-244X2024-10-012411910.1186/s12888-024-06152-wHow accurately can supervised machine learning model predict a targeted psychiatric disorder?Haitham Jahrami0Amir H. Pakpour1Waqar Husain2Achraf Ammar3Zahra Saif4Ali Husain Alsalman5Adel Aloffi6Khaled Trabelsi7Seithikurippu R. Pandi-Perumal8Michael V. Vitiello9Government Hospitals, Psychiatric HospitalDepartment of Nursing, School of Health and Welfare, Jönköping University, HälsohögskolanDepartment of Humanities, COMSATS University Islamabad, Islamabad Campus, Park RoadDepartment of Training and Movement Science, Institute of Sport Science, Johannes Gutenberg-University MainzGovernment Hospitals, Psychiatric HospitalGovernment Hospitals, Psychiatric HospitalGovernment Hospitals, Psychiatric HospitalHigh Institute of Sport and Physical Education of Sfax, University of SfaxCentre for Research and Development, Chandigarh University, MohaliDepartment of Psychiatry & Behavioral Sciences, University of WashingtonAbstract Background Hoarding disorder (HD) is characterized by a compulsion to collect belongings, and to experience significant distress when parting from them. HD is often misdiagnosed for several reasons. These include patient and family lack of recognition that it is a psychiatric disorder and professionals’ lack of relevant expertise with it. This study evaluates the ability of a supervised machine learning (ML) model to match the diagnostic skills of psychiatrists when presented with equivalent information pertinent to symptoms of HD. Methods Five hundred online participants were randomly recruited and completed the Hoarding Rating Scale-Self Report (HRS-SR) and the Generalized Anxiety Disorder 7-item (GAD-7) scale. Responses to the questionnaires were read by an ML model. Responses to the HRS-SR were then converted into anonymized, random-equivalent texts. Each of these individual texts was presented in random order to two experienced psychiatrists who were independently asked for a provisional diagnosis - e.g.; the presence or absence of HD. In case of disagreement between the two assessors, a third psychiatrist broke the tie. A decision tree classification model was employed to predict clinical HD using self-report data from two psychological tests, the HRS-SR and GAD-7. The target variable was whether a participant had clinical HD, while the predictive variables were the continuous scores from the HRS-SR and GAD-7 tests. The model’s performance was evaluated using a confusion matrix, which compared the observed diagnoses with the predicted diagnoses to assess accuracy. Results According to the psychiatrists, approximately 10% of the participants fulfilled DSM-5 diagnostic criteria for HD. 93% of the clinician-identified cases were identified by the ML model based on HRS-SR and GAD-7 scores. A decision tree plot model demonstrated that about 60% of the cases could be detected by the HRS-SR alone while the rest required a combination of HRS-SR and GAD-7 scores. ML evaluation metrics showed satisfactory performance, with a Matthews Correlation Coefficient of 55%; Area Under Curve (AUC), 79%; a Negative Predictive Value of 76%; and a False Negative Rate of 24%. Conclusions Study findings strongly suggest that ML can, in the future, play a significant role in the risk assessment of psychiatric disorders prior to face-to-face consultation. By using AI to scan big data questionnaire responses, wait time for seriously ill patients can be substantially cut, and prognoses substantially improved.https://doi.org/10.1186/s12888-024-06152-wArtificial IntelligenceDisease AssessmentHealthcare AnalyticsHoarding disordersMachine learning |
| spellingShingle | Haitham Jahrami Amir H. Pakpour Waqar Husain Achraf Ammar Zahra Saif Ali Husain Alsalman Adel Aloffi Khaled Trabelsi Seithikurippu R. Pandi-Perumal Michael V. Vitiello How accurately can supervised machine learning model predict a targeted psychiatric disorder? BMC Psychiatry Artificial Intelligence Disease Assessment Healthcare Analytics Hoarding disorders Machine learning |
| title | How accurately can supervised machine learning model predict a targeted psychiatric disorder? |
| title_full | How accurately can supervised machine learning model predict a targeted psychiatric disorder? |
| title_fullStr | How accurately can supervised machine learning model predict a targeted psychiatric disorder? |
| title_full_unstemmed | How accurately can supervised machine learning model predict a targeted psychiatric disorder? |
| title_short | How accurately can supervised machine learning model predict a targeted psychiatric disorder? |
| title_sort | how accurately can supervised machine learning model predict a targeted psychiatric disorder |
| topic | Artificial Intelligence Disease Assessment Healthcare Analytics Hoarding disorders Machine learning |
| url | https://doi.org/10.1186/s12888-024-06152-w |
| work_keys_str_mv | AT haithamjahrami howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT amirhpakpour howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT waqarhusain howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT achrafammar howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT zahrasaif howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT alihusainalsalman howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT adelaloffi howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT khaledtrabelsi howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT seithikurippurpandiperumal howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder AT michaelvvitiello howaccuratelycansupervisedmachinelearningmodelpredictatargetedpsychiatricdisorder |